Teleoperation

For demonstrations and live driving, I built teleop with a Meta Quest. The apps I tried had no passthrough and left me dizzy, so I wrote my own: instead of an app on the headset, the robot serves a page the Quest's browser opens in AR, so the operator still sees the room, and the page streams both controllers' poses, grip and trigger to the robot over a websocket. Holding a grip engages its arm, and the trigger works the gripper. The headset's page goes dark once it enters AR, so the robot also broadcasts that live telemetry to any other browser pointed at it, and someone at a laptop can watch exactly what the operator is sending. It matches the old SDK's interface, so switching was a one-line change. I built it against the arms in MuJoCo first; at first the loop couldn't keep up, and the bottleneck turned out to be publishing the sim viewer over the network, not the control.

The afternoon teleop first ran on the robot, I led a demo for twenty to thirty executives from Winbond, the memory chip maker, shortly before Winbond's $12 million gift to the UW College of Engineering. I prepared and ran it: the robot drove around the room while I teleoperated both arms and explained how it works.

Two days earlier, Fauna Robotics' team visited the lab, and I gave them the same tour: driving the robot around and talking through how it works.