This project encompasses computer vision, machine learning, 3D printing, and Arduino integration, with the primary goal of enabling robotic arms to replicate the movements of a human arm. The project consists of a PyQt6 application where the user can choose to control the robotic arm either through manual gestures, automatic gestures or voice commands. If the user prefers gesture controls, the application will use the OpenCV computer vision library to capture real-time images using a webcam or smartphone, and relevant information from the user's hand will then be detected by one of our neural networks trained based on the YOLOv11 architecture. However, if the user prefers voice command control, the application will use Google's Gemini 2.0 Flash model, the speech_recognition library and the pyaudio library to record and process the user's voice commands. Regardless of the control method, the movement of the robotic arm, which was 3D-printed, is executed by the Arduino according to serial port commands sent by the program.
roboticArm2.mp4
- Windows 10 or 11
- Webcam or a smartphone
- An Arduino board
- Arduino IDE
Access our releases and download the latest one available.
If you don't have a webcam, you can use the Droidcam app on a smartphone to use the phone's camera as a webcam
- Install the Droidcam app (Android or IOS): Download Page
- Install the Droidcam client on Windows: Download Page
- Then follow the instructions to connect: Connection Guide
Follow this guide to install the Arduino IDE.
Once you installed the Arduino IDE, you can download the zip file of the VarSpeedServo library available here and install it by following this guide. Now use the file "serial-port-command-receiver.cpp" in the Arduino IDE.
The following circuit was simulated using the Wokwi online platform, and considering that the four servomotors are already fitted to the robotic arm, the circuit in this image bellow must be physically assembled.
If you want to save time, use our pin configuration as shown in the circuit:
Gripper -> Pin 2
Lower Arm -> Pin 3
Upper Arm -> Pin 4
Rotating Base -> Pin 5
If you want to change the pins, just make sure the digital pins on the Arduino board receiving the signal jumpers from the servomotors are constant values specified in the file "serial-port-command-receiver.cpp" as "#define something_pin".
For example, if the servomotor that controls the robotic arm's gripper is connected to digital pin 2 on the Arduino board, the definition in the code should be "#define gripper_pin 2".
Select the right serial port within the IDE, compile and send the code to the board.
Note that the Voice control method uses Google's Gemini 2.0 Flash model, and it needs an API key, which you can get here. You will need to login with your google account in order to get the API key.
Now just copy and paste the API key to its place in the configuration window:
Check the COM port that your Arduino board is using inside the IDE and put the same value in "COM Port". The default value is 3.
The default capture device index is 0, if it does not work, try 1, 2, etc.
Don't forget to save before closing the Setup window
Now with the app configured, just select "Start":
The following images are examples that show what approximate positions the hand must be in for detections to be made correctly with this method.
The following images are examples that show what approximate positions the hand must be in for detections to be made correctly with this method.
The following voice commands are used for an ideal detection.
Right, Left, Up, Down, Open or Close
But you can also command through short phrases that suggest any of the above commands, such as "go right", "go up" or "open gripper". As long as the sentence does not take more than 2 seconds to be spoken, it should work correctly.
Supervisor: Helton Maia
Project developed for educational purposes at the Automation and Robotics Laboratory (LAR) of the School of Science and Technology (ECT) of the Federal University of Rio Grande do Norte (UFRN).