AI model on a $3 chip (ESP32)(maxlab.io) We finished a design that allows a plug-and-play solution with TensorFlow Lite and a web server with the UI on ESP32. Here is a video of the process: https://youtu.be/aEZX3JMzwTo I wanted to share the design in case anyone is interested as the camera is aimed at developers who want to play with AI models on the embedded side. |
Sorry for being skeptical, just that it sounds too incredible to believe. I've toyed around with the same chips you're using, OV2640 with ESP32-S3 and at any decent resolution it simply seemed to lack the processing power to run even simple motion detection, let alone anything fancier. Surely it was kind of fine at low-res and it kinda works spotting an elephant in a room, but it was completely incapable for detecting small fast-moving targets (roaches) under less than perfect lighting conditions (bathroom ceiling lights, decent but not overly bright). Best it could do was serving a 1024p@5fps MJPEG stream over a network to a more powerful machine for further processing.
https://youtu.be/6raRftH9yxM?t=436
edit: I will say processing IMU data is probably (a lot) easier than frame by frame of some video
It's just the advertising page that sounds kinda unbelievable: low power, night vision, image analysis on the device, perfect for wildlife monitoring, can detect pests on crops (implying high resolution unless we're talking about deer and rabbits lol), etc etc.
I’m hoping to make a pest detector for my automated hydroponic garden. It seems very tractable at this point.
> It has been tested extensively with many processors based on the Arm Cortex-M Series architecture, and has been ported to other architectures including ESP32.
https://www.tensorflow.org/lite/microcontrollers
Additionally there is an optimized kernel for ESP32: https://github.com/espressif/tflite-micro-esp-examples
Edit: personally, I like the Sipeed MAIX for this application, although the software was pretty annoying when I used it (in 2020).
More NLP based, but here is an article on an effort to build Transformers micromodels to run on embedded devices. The model in this example is under 1MB. Goal would be to ultimately convert this from ONNX to TFLite.
https://neuml.hashnode.dev/train-a-language-model-from-scrat...
You could do some rudimentary AI on the ESP32-S3, but you can't send video at a decent rate. The P4 will have H.264 encoding, MIPI connectors, and a 400mhz dual-core.
MJPEG isn't a great way to stream video, and the AI has to be super small to fit into SRAM. You could use flash over SPI, but that's not the greatest.
You can get some decent results, but not for anything I thought was interesting for actual use.
I'm not sure how much sense it actually makes to run anything onboard unless you want to take a snapshot every few seconds and then spending a while classifying it. I've eventually just resorted to streaming at SVGA and decent framerates and then processing the output on another machine.
Will you be at the TinyML Summit, March 27-29? https://www.tinyml.org/event/summit-2023
https://usefulsensors.com/person-sensor/ https://usefulsensors.com/about/