[ Back to home ]

A robot that learns motor skills from a sentence of natural language.

Type "learn the jab punch." The system searches the web, reasons about the biomechanics, compiles a servo trajectory, and executes the motion on two custom 3-DOF arms. No pretraining, no reinforcement learning, no dataset. Hack the North 2025 Top 32, built in 36 hours.

1. Abstract

A zero-shot pipeline that turns a natural-language command into a precise servo sequence on two custom 3-DOF arms. It searches the web for instruction on the named motion, structures a motion guide with an LLM, compiles a synchronized bilateral trajectory, and executes it on hardware.

The architecture is Kahneman's dual-system framing in silicon. A deliberate System 2 reasoning layer (Cohere over Tavily search results) extracts anatomical landmarks from instructional text and emits a structured motion guide. A fast System 1 execution layer (custom skill compiler + ESP32 servo controller) turns it into precisely timed actuator commands at 60 Hz. The LLM never touches a servo register; the firmware never parses prose. The 36-hour window forced that discipline.

The novelty is the pipeline, not the components: a chain from "jab punch" to a 6-servo bilateral sequence, hardened against the two failure modes that kill text-to-motor systems, hallucinated motion guides and unsafe trajectories.

• Two custom 3-DOF arms, six servos, bilateral synchronization. Dual-arm because most useful motor skills are.
• 10 to 15 seconds from sentence to motion: roughly 3 to 5 seconds search, 4 to 7 seconds LLM reasoning, 2 to 3 seconds compile and execute.
• Invalid motion guides cut from 40% to under 5% and unsafe motions from 15% to under 2% by layered filtering on each side of the split.
• Under 50 ms sync between visualisation and hardware, down from 2 to 3 seconds.
• Top 32, Hack the North 2025. Built end to end in 36 hours, judged live on the floor.

2. Introduction

Teaching a robot a new motor skill traditionally costs thousands of examples and weeks of reinforcement learning. A martial-arts student reads a description, watches a demonstration, and starts trying. The internet already describes essentially every human motion; the question is whether that text can drive servos directly, with no intermediate dataset.

• The cost curve is the wrong shape. Reinforcement learning one bimanual skill on a real robot takes weeks of compute and bespoke reward engineering; a second skill costs most of that again.
• The data already exists. Boxing manuals, dance instructionals, surgical-technique writeups, sign-language references, physical-therapy protocols: structured descriptions of motion, none of them conventional training data.
• LLMs can now bridge text and structure. With the right scaffolding, they extract joint configurations from a paragraph of instruction. The bottleneck is no longer linguistic.

2.1 Why dual-system, not a single LLM

Asking the LLM for servo angles directly fails twice: it hallucinates joint values that are kinematically invalid on the real arm, and the prompting needed to force usable output breaks on small wording changes. Splitting slow reasoning from fast deterministic compilation gives each layer a constraint surface it can hold.

• The LLM emits structure, not commands: a JSON motion guide with phases (preparation, execution, recovery), per-arm joint configurations, timing, velocity profiles, and safety metadata. The schema is the contract.
• The compiler enforces physics. Inverse kinematics, smoothing, velocity profiling, and safety clamping sit downstream of the LLM and catch anything outside the kinematic envelope.
• Failure modes separate. Hallucinated guides fail at cross-source agreement; unsafe trajectories fail at simulation rehearsal.

2.2 What is actually new

Cohere, Tavily, the ESP32, and the arm geometry are off the shelf. The contribution is the pipeline that holds them together against the two failure modes text-to-motor systems usually die on.

• Structured motion guides as the intermediate format, so the compiler never re-prompts the LLM.
• Multi-stage hallucination filtering: relevance scoring, anatomical landmark extraction, cross-source agreement, and a feasibility check against the kinematic envelope. Each stage is small; the composition gets 40% to under 5%.
• Synchronized bilateral execution. Both arms' commands share one batched stream, so phase alignment holds under socket jitter.

Section 3 is the engineering behind those three commitments.

3. System

3.1 Reasoning layer (System 2)

• Tavily Search queries the web for instructional content. Query engineering rewrites the user's phrase into a search-optimised form ("jab punch" becomes "boxing jab technique biomechanics step by step"), so results skew toward structured instruction, not match commentary.
• Cohere agent scores relevance, extracts anatomical landmarks (shoulder rotation, elbow extension, wrist position), maps each to the robot's joint configuration, and cross-references multiple sources to catch single-source artifacts.
• Output is the JSON motion guide, timed in milliseconds, with safety metadata flagging any out-of-envelope joint target.
• Filtering: roughly 40% of first-pass guides fail at least one of the four stages; only survivors reach the compiler.
• Ablation: dropping cross-source agreement alone brings the invalid-guide rate back up to roughly 20%. The composition matters more than any single filter.

3.2 Execution layer (System 1)

• SkillCompiler turns the guide into a servo command stream. Inverse kinematics resolves joint targets, cubic-spline smoothing softens phase transitions, velocity profiling matches per-phase timing, and safety clamping enforces 0 to 180 degree limits and a capped angular velocity.
• No-overlap zones between the arms are forbidden joint-space regions; any trajectory crossing one is rejected before transmission.
• RobotControlGenerator serialises the stream into a binary protocol for the ESP32. Batched updates pack four servo positions per message at 15 Hz, the equivalent of 60 Hz per-servo control with a fifth of the wire traffic.
• ESP32 firmware receives commands over Socket.IO, drives the six servos at microsecond timing precision, and returns timestamped acknowledgements so the visualisation stays in lockstep.
• Ablation: removing simulation rehearsal alone brings the unsafe-motion rate back up to roughly 8%. Static clamping plus dynamic rehearsal reaches under 2%.

3.3 Frontend and visualisation

• Next.js + React chat interface renders the reasoning trace live as the LLM streams its motion guide.
• Three.js 3D viewer renders predicted servo positions in blue against hardware-confirmed positions in green. Visible drift flags command-protocol bugs immediately.
• Arduino UNO + TFT display shows an animated monkey avatar tied to robot state (idle, active, error): cosmetic, but a useful at-a-glance status signal on the demo floor.

3.4 Synchronization

• Timestamp-based command protocol with explicit acknowledgements cut visualisation-to-hardware drift from 2 to 3 seconds on the first naive implementation to under 50 ms.
• Socket.IO heartbeats every 500 ms. Three consecutive failures trigger a reconnect; on connection loss the ESP32 enters a known safe state rather than holding the last commanded position.
• Priority queue for emergency-stop and position-reset commands, which bypass the batched stream so a safety-critical signal never waits behind routine updates.

4. Evaluation

Qualitative and demo-driven, measured on the build's own protocol rather than an external benchmark for natural-language robotic control.

• Live demo at Hack the North 2025. Jab punch, wave, beckon, and a celebratory arm raise, learned and executed on the spot from natural-language input with no previous training on any of them.
• 10 to 15 seconds end to end across all four skills.
• Invalid motion guides: 40% to under 5%, measured across the broader set of motions tested during the build, not just the four demoed.
• Unsafe motions: 15% to under 2% after static clamping plus simulation rehearsal.
• Under 50 ms sync, validated by the timestamped acknowledgement flow.

Both filtering gains are measured against the build's own first-pass baseline; no widely accepted natural-language robotic-control benchmark exists for hackathon-scale dual-arm setups. The four skills are an existence proof of the end-to-end loop, not a generalisation claim. Read the numbers as evidence that the dual-system architecture closes the text-to-motor loop under hackathon constraints, not as a deployable robotic-control system.

5. Conclusion and Implications

The internet contains the knowledge robots need to learn new tasks; the pipeline that gets it onto silicon is the contribution. The dual-system split generalises wherever the bottleneck is reprogramming overhead, not motor precision: assistive healthcare robotics that adapts to a new patient's range of motion in minutes, surgical training rigs that demonstrate a new technique without bespoke RL, industrial prototyping where teaching a task costs a sentence. The safety consequence is the same in each: a robot that learns from prose, with hallucination filtering and trajectory rehearsal as first-class layers, is the only kind that can be trusted to act on text it has never seen.

5.1 Limitations and future work

• Open-loop execution. A slipped servo or obstruction surfaces only at the acknowledgement timeout, not from the world. Next: a vision-language model scores each executed motion and the compiler iterates ("pick up the red cup" first), plus pose estimation on video demonstrations for richer phase timing.
• No persistent skill library. Every request re-queries the web and re-runs the LLM. Next: cache successful guides and compose new skills from old ones ("wave plus forward step").
• Single providers. A Cohere or Tavily outage takes the system offline; the architecture supports fallback, but it was not wired up during the event. Next: automatic multi-provider failover.
• Limited motion range. The demo motions are upper-body and bilateral. Fine motor skills, torso or leg coordination, and contact-reactive motions (grasping, pushing against resistance) are beyond the current 3-DOF arms, which have no end-effector force feedback.
• Two arms only. Next: multi-robot coordination over a shared command bus, so one instruction drives synchronised motion across more than two arms.

6. Acknowledgments

Built at Hack the North 2025, recognised as a Top 32 Finalist. The reasoning layer relies on Cohere; the search layer relies on Tavily. Hardware control runs on the standard Arduino IDE toolchain over Socket.IO for the host link.