Download
Abstract
Training state-of-the-art imitation learning methods, like diffusion policy and ACT, has become easier with recent open source libraries like Huggingface’s LeRobot. But building the pipeline to teleoperate an arbitrary pair of robot arms in real time, recording demonstrations for imitation learning in an intuitive way, requires some expertise to set up. I built a browser app to make this easy: WebXR-based handtracking, connecting via WebRTC to you laptop, which solves the inverse kinematics for arbirary URDFs in WASM in the browser, exporting trajectories containing movements and webcam-vision that are ready for imitation learning. All you need to write is a little python server forwarding joint angles from localhost to your robot, which may server custom URDFs to the webapp.
Demo
Here’s a demo of the teleop stack applied to a pair of MyCobot 280 M5 arms:
The MyCobot arms unfortunately have severe latency, which reduces your dexterity. I highly recommend investing in better hardware. For reference, the response time of the teleop stack itself is virtually unnoticeable:
