About This Camera

This camera, named "TIME LEAP CAM," is designed for long-interval time-lapse photography and extended monitoring. The name reflects its ability to "leap" through time by capturing images over long periods. While there are many cameras with monitoring and time-lapse capabilities, none can adapt to various conditions and make intelligent judgments based on those conditions.

One challenge was the need for communication to send images. To keep the camera power-efficient and cost-effective, I chose the ESP32S3 SoC, which supports WiFi communication and HTTPS (SSL) protocols. The camera uses WiFi to connect to the internet and employs OpenAI's GPT-4o to analyze the images. If certain conditions are met, the camera sends notifications via LINE. For example, it can notify if a person has fallen if a pet's food bowl is empty, if plants appear unhealthy, if a window is open, or if it looks like it might rain. This eliminates the need for constant human monitoring. The LINE notifications include relevant images, and the conditions for notifications can be easily set using natural language through a configuration screen.

The camera is battery-powered and enters DeepSleep mode when not capturing images to conserve power. Captured images are stored on an internal 64GB eMMC. There are 11 folders available for organizing the stored images. The camera is designed to be water-resistant for outdoor use, though it is not waterproof. Charging is done via a Qi wireless charger, and settings can be configured through a web browser on a smartphone or PC over WiFi. If the camera is in sleep mode, touching a pad on the back activates the web server for configuration access. The camera automatically returns to sleep mode after 5 minutes of inactivity.

To use the image analysis and LINE notification features, an OpenAI API key, CloudFlare Images service access ID and API key, and LINE Messaging API account ID and access token are required. Internet communication via WiFi is also necessary. For time-lapse photography only, WiFi communication is not needed.

Service IDs and access keys can be obtained from the respective service websites:

- OpenAI API: https://openai.com/index/openai-api/

- CloudFlare Images: https://developers.cloudflare.com/images/about

- LINE Messaging API: https://developers.line.biz/en/docs/messaging-api/

Features of This Camera

• Auto-focus
• Long-interval shooting
• Power-saving DeepSleep mode (less than 440uA during sleep, 1.5mW)
• Qi wireless charging with automatic stop when fully charged
• Water resistance
• Image analysis and LINE notifications using OpenAI GPT-4o/GPT-4-Turbo models
• Natural language notification condition settings (multi-language input supported)
• Touch-activated web server and automatic sleep mode
• Time synchronization via NTP (automatic adjustment when WiFi is connected) and RTC for timekeeping

Specifications of This Camera

• Resolution: Up to 2592x1944 (QSXGA)
• Sensor: OV5640 5MP sensor
• SoC Module: ESP32-S3-WROOM-1-N16R8 (240MHz, 512KB SRAM, 16MB Flash, 8MB PSRAM)
• Battery: 3.7V LiPo 1100mAh
• Storage: 64GB eMMC
• Communication: WiFi 2.4GHz
• Charging: Qi wireless charging
• Dimensions: 67(W) x 78(H) x 66(D) mm
• 
  

Notes

Water Immersion: The camera is not designed to withstand full submersion in water.

API Keys Required: To use the image recognition and notification features, obtain API keys from OpenAI, CloudFlare Images, and LINE Messaging API.

Internet Connection: Required for using advanced features like image recognition and notifications. Time-lapse functionality is available without WiFi.

How it works

The TIME LEAP CAM operates by capturing images at pre-set intervals or specific times. These images are stored on the internal 64GB eMMC storage. During each capture, the camera performs AutoFocus to ensure clear images. The captured images are saved in one of the 11 available tracks, which can be configured as needed.

In monitoring mode, the captured images are sent to OpenAI's GPT-4o or GPT-4-Turbo along with a prompt (e.g., "If there is somebody, add 'NOTICE' in the reply and reason. If there is nobody, just reply 'NONE'."). The AI analyzes the image content based on the prompt. If the response contains "NOTICE," it indicates that the image meets the specified conditions. If the response is "NONE," it does not.

When "NOTICE" is detected, the image is uploaded to CloudFlare Images to generate a URL for the image. This URL, along with a notification message, is then sent to the LINE Messaging API. The designated LINE account receives the notification along with the image.

Notification conditions can be set using natural language, and the system supports multiple languages as long as they are supported by GPT-4o.

The camera is powered by a 3.7V LiPo 1100mAh battery, which is charged using a Qi wireless charger. Simply placing the camera on the Qi charger initiates the charging process, and charging stops automatically once the battery is full.

Configuration and image review can be done via a smartphone or PC browser over WiFi. Users can set parameters such as capture intervals, capture times, and resolution. Accessing the camera's IP address (e.g., http://192.168.2.100) opens the settings interface. The IP address is assigned by the DHCP server of the connected WiFi network and can be checked in the camera's startup console after flashing the firmware. Currently, the IP address, WiFi SSID, password, OpenAI API key, and CloudFlare access ID and API key must be set in the source code's cfg.toml file.

When the camera is in sleep mode, touching the back of the camera activates the web server, allowing access to the settings interface. If there is no interaction for 5 minutes, the camera automatically returns to sleep mode.

If capturing image intervals are over 90 seconds, the camera will be in deep sleep after capturing an image. If the duration is less than 60 seconds, the camera won't enter sleep mode. This is because the ESP32 Deep Sleep function runs an initialization sequence upon waking up, which takes about 30 seconds to complete.

The camera synchronizes its time using NTP when connected to WiFi and maintains the time using an RTC.

Design a Schematic

This is Qi Reciever schematic. I used BQ51013BRHLR for the control IC. AC1 and AC2 are connected to a receiver coil WR483265-15F5-G. If you connect the USB to a Host PC, this camera can charge to the battery from USB power.

ESP32-S3-WROOM, eMMC, and Camera Interface.

To reduce consumption power, eMMC and Camera power are controlled by ESP32-S3. By this control, power consumption is less than 1.5 mW. Pull-up registers for the eMMC line are 100 Kohms. It will suppress power consumption, too.

A camera interface has 24 pins connector. Pin 23, and 24 are used for the autofocus camera motor power. However, these pins are different from other OV5640 module products. I used the Seeed OV5640 Camera for ESP32S3 Sense. Other OV5640 modules sometimes are that pin23 is VDD, and pin24 is GND. It will not work.

PCB Layout

I used 4 layer Custom PCB.

I ordered PCBWay to manufacture my board. Click the "Add Gerber File" button, and you can upload the Gerber file.

When ordering the PCB, the parameters can be almost default values, but "Min hole size" should be 0.25mm, and "Immersion gold (ENIG)" should be selected for "Surface finish". The reason for this is that the minimum Via Hole is 0.25mm, and if the eMMC pad is not "Immersion gold (ENIG)", the solder on the pad will be slightly raised and cannot be soldered properly without misalignment.

After the order, I received boards about 1 week later.

After assembly, all components are installed.

Install Software

The software for this camera is written in Rust. The compilation steps for Ubuntu 22.04 are detailed in the GitHub README. Please follow the steps below to install the software:

1. System Update and Package Installation

Update your system and install the necessary packages using:

sudo apt update && sudo apt -y install git python3 python3-pip gcc build-essential curl pkg-config libudev-dev libtinfo5 clang libclang-dev llvm-dev udev libssl-dev python3.10-venv

2. Rust Installation

Install Rust programming language and Cargo package manager:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

After installation, activate Rust by sourcing the environment:

. "$HOME/.cargo/env"

3. Additional Tools Installation

Install the following Rust tools:

ldproxy

espup

cargo-espflash

Use the following commands:

cargo install ldproxy cargo install espup cargo install cargo-espflash

4. ESP Environment Setup

Run the following command to install and update the Espressif Rust ecosystem:

espup install espup update

Set up environment variables:

. ./export-esp.sh

5. Udev Rules Configuration

Configure udev rules for device permissions:

sudo sh -c 'echo "SUBSYSTEMS==\"usb\", ATTRS{idVendor}==\"303a\", ATTRS{idProduct}==\"1001\", MODE=\"0666\"" > /etc/udev/rules.d/99-esp32.rules' sudo udevadm control --reload-rules sudo udevadm trigger

6. Clone Repository

Clone the TimeLeapCam repository:

git clone https://github.com/hnz1102/timeleapcam.git cd timeleapcam/code/

7. Setting WiFi SSID, Password, etc.

Change the following configuration file.

cfg.toml

[timeleapcam] wifi_ssid = "<Your AP SSID>" # Set your AP SSID wifi_psk = "<Your AP Password>" # Set your AP Password http_server = "" # NOT USED resolution = "8" track_id = "0" timezone_offset = "9" auto_capture = "false" idle_in_sleep_time = "300" duration = "0" api_key = "<API KEY for openAI>" # Set your OpenAI API Key model = "gpt-4o" query_openai = "false" query_prompt = "If is there somebody, Add in the reply as 'NOTICE' and reason. If is nobody, Just reply is 'NONE'." post_account = "<Your LINE Account ID>" # Set your LINE Account ID post_access_token = "<Access Token for LINE Message>" # Set your LINE Access Token storage_account = "<Your Cloudflare Account ID>" # Set your Cloudflare Account ID storage_access_token = "Your Cloudflare Access Token>" # Set your Cloudflare Access Token post_message_trigger = "NOTICE"

8. Build and Flash

Build the project and flash it to your device:

cargo build --release cargo espflash flash --release --monitor

9. Run the Camera

After flashing, the camera will boot up. The console will display startup messages. The first boot may take a few minutes to format the eMMC. Once completed, an IP address will be displayed:

I (6520) esp_netif_handlers: sta ip: 192.168.2.191, mask: 255.255.255.0, gw: 192.168.2.1

Open a web browser and navigate to the displayed IP address (e.g., `http://192.168.2.191`). The configuration interface will allow you to set the capture interval, capture time, resolution, and other settings.

Ensure the camera is functioning correctly by checking the console output for messages indicating successful initialization:

I (9550) camera: Detected OV5640 camera I (9550) camera: Camera PID=0x5640 VER=0x00 MIDL=0x00 MIDH=0x00 I (10250) cam_hal: buffer_size: 16384, half_buffer_size: 1024, node_buffer_size: 1024, node_cnt: 16, total_cnt: 960 I (10250) cam_hal: Allocating 983040 Byte frame buffer in PSRAM I (10250) cam_hal: cam config ok I (10270) ov5640: Set PLL: bypass: 0, multiplier: 200, sys_div: 4, pre_div: 2, root_2x: 0, pclk_root_div: 2, pclk_manual: 1, pclk_div: 4 I (10270) ov5640: Calculated XVCLK: 5000000 Hz, REFIN: 2500000 Hz, VCO: 500000000 Hz, PLL_CLK: 50000000 Hz, SYSCLK: 12500000 Hz, PCLK: 3125000 Hz I (10310) timeleapcam::capture: Capturing Frame Thread Start... I (10310) timeleapcam::monitoring: Query thread started I (10310) timeleapcam::autofocus: Autofocus PID: 0x5640 I (10320) timeleapcam: Resolution changed: 21 -> 8 I (15270) timeleapcam::autofocus: Waiting for autofocus to start... I (15270) timeleapcam::autofocus: Autofocus initialized I (15290) timeleapcam::autofocus: AutoFocus Mode Enabled I (15290) timeleapcam::autofocus: AutoFocus Triggered I (25310) timeleapcam::autofocus: AutoFocus Timeout I (25320) ov5640: Set PLL: bypass: 0, multiplier: 180, sys_div: 4, pre_div: 2, root_2x: 0, pclk_root_div: 2, pclk_manual: 1, pclk_div: 4 I (25320) ov5640: Calculated XVCLK: 5000000 Hz, REFIN: 2500000 Hz, VCO: 450000000 Hz, PLL_CLK: 45000000 Hz, SYSCLK: 11250000 Hz, PCLK: 2812500 Hz I (25360) timeleapcam::autofocus: AutoFocus Mode Enabled I (25360) timeleapcam::autofocus: AutoFocus Triggered I (29490) timeleapcam::autofocus: AutoFocus Done

By following these steps, you will have successfully installed and configured the software for your Time Leap Cam.

Container Design

This camera is housed in a case created using a 3D printer. The case is designed with a certain level of water resistance to accommodate outdoor use, although it is not waterproof against submersion. At the base of the case, a coil is installed to receive power from a Qi wireless charger, and the battery is placed above this coil. The PCB is then mounted above the battery, with the camera portion protruding from the front.

A lid is attached to the base case to secure the internal PCB and camera components. Inside the case, there is a USB port for software writing and a screw that functions as a touch switch. A cover case is then placed over this assembly. The cover case includes a round acrylic plate to protect the camera lens while allowing it to be visible.

This case was created using a 3D printer. The STL data for the case can be downloaded from the following link. GitHub

Conclusion

In conclusion, the "TIME LEAP CAM" is an auto-focus camera designed for long-interval time-lapse photography and extended monitoring. It leverages the capabilities of OpenAI's GPT-4o to analyze captured images and send notifications via LINE when specific conditions are met, such as detecting a person falling or a window being open. The camera operates on a low-power ESP32S3 SoC, supports WiFi communication, and uses Qi wireless charging. It features deep sleep mode for power conservation, a 64GB eMMC for storage, and a robust design suitable for outdoor use. The camera's settings can be easily configured through a web interface accessible via WiFi. This device addresses the need for automated monitoring without constant human supervision, making it a valuable tool for various applications, including elder care and environmental monitoring.

Finally, I would like to thank PCBWay for their support of the custom PCB board.

LICENSE

This source code is licensed under MIT. Other Hardware Schematic documents are licensed under CC-BY-SA V4.0.

This project uses the following libraries and resources:

https://github.com/jlocash/esp-camera-rs

https://github.com/0015/ESP32-OV5640-AF

These libraries are licensed under each library's respective licenses.