[ Back to home ]

Real-time gesture control of a 3D-printed robotic hand.

21 MediaPipe hand landmarks per frame, a finger-state classifier that reduces them to a five-bit command, a tight serial protocol, and five servos on an Arduino driving a custom 3D-printed hand. Webcam in, mirrored motion out, no GPU.

1. Abstract

A real-time gesture-control loop that maps a user's finger positions onto a custom 3D-printed robotic hand. MediaPipe Hands extracts 21 landmarks per frame, a per-finger classifier reduces them to a five-bit up-or-down command, a deterministic serial protocol carries it to an Arduino, and five servos with antagonistic cable actuation drive a printed PLA hand that mirrors the gesture.

The implementation is deliberately spare. No GPU. No deep model beyond MediaPipe's bundled networks. No continuous angle tracking, just five binary fingers. The serial buffer flushes every frame, so the firmware always acts on the latest hand state rather than queued stale commands.

The point is the loop, not any single piece: vision, classification, serial, firmware, and actuation chained end to end on commodity hardware. This early project is where writing each layer simply enough for the next to trust became the rule for everything after.

• 21 hand landmarks per frame via MediaPipe Hands at 30+ FPS on a standard CPU. No GPU required.
• Binary five-finger classification from tip-versus-joint landmark comparisons, which stabilised the loop.
• Tight serial protocol. $TTTTT with five binary digits at 9600 baud.
• Five servos, one per finger, with antagonistic cable plus elastic-band actuation that mimics the flexor and extensor pairing in a biological hand.
• External 5 V supply with 100 microfarad decoupling near each servo cluster, because the Arduino regulator alone cannot source the roughly 2.5 A peak when all five servos move simultaneously.

2. Introduction

Robotic teleoperation usually means joysticks or expensive motion-capture rigs, which make the user translate gestures into something the rig understands. That translation is the friction. MediaPipe brings precise hand tracking to a webcam and Arduino brings cheap servo control to anyone with a soldering iron, removing most of the cost barrier that kept gesture-controlled robotics in research labs.

• Gesture is the most natural control surface for a hand. Mapping fingers to fingers preserves the user's mental model. The user does not learn the rig; the rig follows the user.
• The pieces are now commodity. Webcams, MediaPipe, Arduino, hobby servos, and PLA-printed parts are each cheap and well-documented in isolation. Stitching them into a working loop is the work.

2.1 Why binary state, not continuous angles

The first attempt mapped continuously tracked joint angles linearly to servo positions, and the result was visibly jittery. MediaPipe's landmarks carry a few pixels of frame-to-frame noise, which became a few degrees of servo command at the camera's rate, and the servos could not keep up with rapid small commands. Collapsing each finger to a binary up-or-down state, decided by comparing tip and joint landmarks, removed the noise floor entirely. The vocabulary shrinks to 32 distinct hand poses instead of a continuum, but every pose is recognisable and reproducible.

• Noise is the architecture's problem. A downstream classifier cannot fix upstream noise; a discrete state lifts the signal above the noise floor at the source.
• The actuator's slew rate is the second constraint. Hobby servos move roughly 60 degrees per 0.1 seconds. A continuous command stream above that rate is wasted; a discrete stream at that rate is exactly what the hardware can follow.
• The gesture set stays useful. The 32-pose space covers what most gesture-controlled interfaces actually need.

2.2 What is actually new

MediaPipe is from Google. Arduino is Arduino. The cable-actuated hand geometry is borrowed from the open-source prosthetics community. The contribution is the end-to-end loop, closed cheaply and reliably on commodity hardware.

• End-to-end loop on a webcam and an Arduino. No GPU, no specialised tracking hardware, no industrial servos.
• Antagonistic cable plus elastic-band actuation. One servo pulls; the elastic returns. Flexor-extensor pairing without two servos per finger.
• Power discipline as a first-class layer. The hardware iteration that taught it is part of why the project ships at all.

Each section below is one layer in the loop.

3. System

3.1 Vision pipeline

• MediaPipe Hands runs a two-stage palm detector then landmark localisation, producing 21 keypoints per hand with (x, y, z) coordinates per frame.
• Mirror flip on capture so the user's right hand controls the rig's right side, removing the cognitive cost of mirrored teleoperation.
• BGR to RGB conversion before MediaPipe (OpenCV reads BGR, MediaPipe expects RGB). Landmark complexity is level 1, the speed-versus-accuracy midpoint; the classifier needs landmark relationships, not millimetre precision.
• Debug overlay draws the bounding box, landmarks, and finger states on the feed. Off during demos.

3.2 Finger-state classifier

No trained model, just direct comparisons that exploit how the 21 landmarks are laid out.

• Thumb compares landmark 4 versus 3 horizontally. It sits perpendicular to the other fingers, so their vertical comparison does not apply.
• Index, middle, ring, and pinky compare tip-landmark y with PIP-joint y. Tip above joint means extended; tip below means flexed.
• Intermediate poses default to "down" to prevent flicker. Half-extended fingers are rare in normal gestures, and rounding to flexed is more stable than oscillating.

3.3 Serial protocol

Five-character payload $TTTTT, where each T is the ASCII digit 0 or 1. The $ delimiter lets the firmware locate message boundaries even on partial reads.

• Fist is $00000. Open hand is $11111. Peace sign is $01100. Readable in the serial monitor while debugging.
• 9600 baud. The minimal payload keeps latency well under the point where mirroring stops feeling responsive.
• Buffer flush per transmit. An earlier iteration queued commands and lagged visibly when the user moved rapidly. Flushing was the fix.

3.4 Firmware

• Arduino Uno reads the serial port continuously, watches for the $ delimiter, parses the next five characters, and writes 0 or 180 degrees to each servo via PWM.
• Servos on digital pins 2 through 6, one per finger, hardcoded because the hand's cable runs are physically wired to specific servos.
• External 5 V supply rated for at least 3 A, with 100 microfarad decoupling capacitors near each servo cluster to suppress brown-out resets from the current spike. Learned by failing first: the early build kept rebooting mid-gesture until the caps went in.

3.5 Mechanical hand

Articulated finger segments designed in Fusion 360 and printed in PLA. Each finger has its own cable run from a palm-mounted servo, with elastic bands providing return force. One degree of freedom per finger keeps the mechanics simple and the gestures recognisable from across a room.

4. Evaluation

Evaluation is qualitative and demo-driven, measured on the build's own setup, not an external teleoperation benchmark.

• 30+ FPS detection on a standard CPU with the integrated webcam; 60+ FPS observed on a CUDA-capable GPU.
• Sub-perceptible end-to-end latency on the standard test gestures. Mirroring feels live, the only useful subjective threshold for teleoperation.
• Recognisable gestures with no per-user calibration. Fist, open hand, peace sign, point, and thumbs-up classify reliably across hand sizes and skin tones without retraining MediaPipe.
• Primary failure modes identified. Poor lighting (histogram equalisation helps), partial occlusion (temporal smoothing fills brief drops), and motion faster than the servo slew rate, which is the hardware ceiling.

Frame rates are from the development laptop's integrated camera and CPU. The gesture claim is qualitative across a few dozen hand poses, not a quantitative accuracy figure. Read this as an early end-to-end systems exercise, not a gesture-recognition benchmark.

5. Conclusion and Implications

The same loop generalises far beyond hand-shadow puppets.

• Prosthetic control. Amputees translate residual gestures into actuator commands without expensive EMG electrodes.
• Hazardous-environment teleoperation. Lower operator training overhead matters more than peak precision.
• Surgical training rigs. Natural hand input replaces master-and-slave manipulators.
• Accessible robotics education. Students wire a complete vision-to-actuator pipeline in a weekend.

In each case the safety-critical consequence is the same: a teleoperation loop on commodity hardware, with each layer simple enough that failure modes are observable rather than buried, can be debugged by the people who deploy it.

5.1 Limitations and future work

• Binary state per finger. The 32-pose vocabulary misses partial flexion (a precision pinch, a relaxed grip). Next: continuous angle control from MediaPipe's (x, y, z) with temporal smoothing, and a stereo or depth camera (RealSense) for true 3D joint reconstruction.
• Open-loop control. A jammed cable or slipped servo goes undetected. Servo current sensing and fingertip pressure sensors could close the loop back to a haptic glove.
• Single hand, one degree of freedom per finger. No wrist control, thumb opposition, or finger spread; wrist and thumb opposition come next.
• Lighting dependence. MediaPipe degrades in low light, which propagates straight into classifier flicker.
• Tethered to the host by the serial cable. An ESP32 link over Wi-Fi or Bluetooth would unlock remote teleoperation.

6. Acknowledgments

Hand-landmark tracking provided by Google MediaPipe Hands. The cable-actuated 3D-printed hand geometry draws on conventions from the open-source prosthetics community. Built independently.