Controlling digital devices with hand gestures is so cool and you can control things like media, APPs, etc. but controlling physical objects with hand gestures is much cooler.

In this project with the help of computer vision I'm controlling printed photo with hand gestures.

you can use this idea and code to control any physical objects with hand gestures.

this project have two parts:

first part is a python code that runs on the computer.

and second part is hardware, including Arduino board and 3 servo motors.

in python code with the help of opencv and mediapipe we can trace finger motions and based on finger motions we send commands to arduino board with serial port and Arduino controls servo motors.

For running the python code you need PyCharm software

before running the code you should install:

opencv

mediapipe

Serial

OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library.

OpenCV was built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception in the commercial products.

MediaPipe offers open source cross-platform, customizable ML solutions for live and streaming media. 

In this unique augmented reality device, not only is the object being controlled in the real world but the gestures are being monitored there as well, thanks to a computer vision system watching my hand which is running OpenCV. The position data is fed into an algorithm which controls a physical picture mounted on a slender robotic arm.

Now, when I “pinching to zoom”, the servo attached to the picture physically brings it closer to or further from his field of view. He can also use other gestures to move the picture around.

While this gesture-controlled machine is certainly a proof-of-concept, there are plenty of other uses for gesture controls of real-world objects. Any robotics platform could benefit from an interface like this,