Collaboration

OpenDubs runs on an NVIDIA Jetson Thor, so the first job was getting it ready for the whole team. I flashed JetPack onto it, got its IO talking to the rest of the robot, and set up the Docker environment we all develop in: a ROS 2 image with the build tools, Makefile and VS Code tasks to build and debug inside it, and per-person git identities so four people can commit from one shared machine.

The first version of the base drove its four mecanum wheels through a Pixhawk running ArduPilot, over MAVROS; it has since dropped the Pixhawk. With another intern, I brought up that drive code, then added what the rest of the stack needed: a URDF and an SDF with collision geometry for RViz and Gazebo, and a service that records topics to rosbags. Every test after this was a bag I could replay.

The base has since been rebuilt by others in the lab (I didn't work on it), without the Pixhawk, and carries everything else: a 51.2 V LiFePO4 battery, an emergency stop on the deck and a status display, all wired into one aluminum frame, with the arms, the Thor and the lidars riding on top.

Advanced Robotics at UW competes in ARC, DJI's robotics competition for universities in North America (RoboMaster North America until 2025). Every season the team designs and builds a fleet of eight robots, most of them new (this year's reveal), for what the team calls a deathmatch somewhere between paintball and League of Legends. Operators drive their robots through an onboard camera and fire 17 mm and 42 mm projectiles at the pressure-sensitive armor plates on the other team's robots, while fighting king-of-the-hill style for a capture point in the middle of the field, in matches that are each a best of three. The engineer can't shoot: it collects minerals to trade for coins that buy supplies during the match. The sentry is the only fully autonomous robot on the field, finding its own way around and aiming on its own. ARUW won the competition every year from 2021 to 2024. At the 2026 ARC Championship at Purdue we placed third in the 3v3, second in the dart event and third in the drone event; for it I worked on the engineer's odometry and path tracking, and since then I've moved to the sentry's localization and navigation.

In online RL an arm practises a task over and over, so after every try something has to put the peg back where the task starts. I built that reset for the peg-insertion task. A teammate and I designed the tagging together, and they wrote the code that reads it: ten AprilTags on the peg's faces, from which the scene camera solves the peg's full pose. I calibrated each camera with a ChArUco board, first its lens (the intrinsics), then its pose relative to the arm (the extrinsics). For the second, a calibration script has the arm set the tagged peg down at a grid of spots on the board and fits the arm's joint zeros and the camera's pose to where the arm thought the peg was against where the camera saw it. With that, the reset finds the peg anywhere in the workspace, works out how it is lying, and plans every waypoint of the pick and the place before the arm moves, choosing the grasp the arm can reach with the least travel within its joint limits. It then checks where the peg landed. This lets the teammate's automatic success detection score each try, so the arms can run online RL on their own. The reset has run on a real arm; the clip shows the same planner in simulation.

A VLA can look great near its training setup and fall apart a few centimeters away, so the hard part is deciding what a fair test is. We picked eight tasks across six kinds of challenge and changed the objects' placement and the wording on each of ten rollouts. We randomized the placements within a set workspace before running any evaluation, and every method ran the same ones. If I ran it again, the biggest thing I'd change is the lighting: we didn't control it well, and the models performed noticeably differently early in the morning than late at night.

Success alone also hides a lot, so in the main benchmark each rollout was scored three ways: did it succeed, what fraction of the goals were met, and how far along a written rubric it got. Separately from that benchmark, we ran an ablation on wording alone: one task with the apple held still and only the instruction changed, fifteen phrasings run once each. π0.5 succeeded on "Grab the apple and set it on the pan" and failed on "Take the apple and place it onto the pan." It was sensitive to how specific the prompt was: given an underspecified one like "Put the apple away," the arm wandered without targeting the apple before eventually turning to the task, and an overly specified prompt degraded it the same way. It struggles when the task's intent is missing or buried.

MolmoAct2 was strong on the banana but struggled with the other pick-and-place tasks, which suggests it's fragile to object type, and on the harder tasks it often misread the prompt entirely, pressing the top of the cabinet instead of closing the drawer. It also disengaged early: in the two-object task it moved one object and withdrew, and in the cluttered box it picked the wrong object and never came back for the right one. Across all of it, the VLAs failed in the same few ways: sensitivity to how the prompt was worded, dependence on the object (π0 never got the baseball into the bucket), and poor spatial reasoning. They ignored vertical clearance, like the height of a barrier or how high to lift a cloth before setting it aside, and when a task called for a specific procedure they defaulted to another one that couldn't finish it, grabbing the drawer's lip instead of pushing it closed.

For localization and a map, the base carries two RPLIDAR A1 2D lidars, one at the front and one at the back. Bringing them up, with the team, was mostly fighting USB: the two units have identical USB IDs, so udev rules name each by the hub port it's plugged into, and they power up a few seconds apart so neither times out.

Mapping and Nav2 expect one scan, so I wrote a node that merges the two. The two scans never line up on their own: nothing guarantees the lidars spin at exactly the same rate or finish a revolution at the same moment, and each reaches the Thor with its own latency. So the node pairs scans by their timestamps (any two within 0.1 s), looks up each scan's transform into the robot's frame at that scan's own timestamp, resamples both to the finer of the two angular resolutions, and keeps the nearest return in each direction of one 360° scan, stamped halfway between the two and dropping hits on the robot itself. On top of it, I set up slam_toolbox and Nav2, which drove the robot point to point in simulation.