Experiment design

The question wasn't how accurate the Odin is but whether the sentry could rely on it alone, and one accuracy number can't answer that: the nav stack needs to know how the error behaves, whether it grows, and what happens when something goes wrong. So I designed the October 3 campaign around those questions (figure-8 laps, a long hold, repeated relocalizations and covered-sensor trials, all against mocap) and fit a small model to it: the true pose, plus an offset drawn each time the Odin relocalizes into its map, plus first-order Gauss-Markov noise: correlated over time, but decaying.

Splitting the error this way was the key decision. The offset from each relocalization turned out to be the largest term, which one overall noise figure would have blurred, and modelling the noise as decaying showed the error stays bounded instead of drifting. To check the model wasn't just describing the data it was fit to, I refit it without one session, simulated it many times along that session's real path (a Monte Carlo check), and asked whether the real error fell where the simulations said it would. It did: its real error there, 5.1 cm horizontal and under half a degree, matched the prediction on all six axes (roll came in just under its predicted range, still within the test). That made the recommendation clear: the Odin can be the sentry's only localization, kept on the turret major, where it swivels independently of the turret and the chassis. We tried it in several mount locations and found the occlusions made little difference. The nav stack only has to handle a rare wrong relocalization, and the output pausing while the sensor is covered.

The sentry localizes with an Odin1, a lidar-inertial sensor that relocalizes into a prebuilt map. Before the team decided whether the whole localization system could rely on it alone, we wanted to know how accurate it is and where it is weak. So I built a test platform that measures it against the motion capture room, whose OptiTrack tracks the robot to a fraction of a millimeter. It starts and stops the sensor, records every run, guides whoever is driving with live mocap feedback, and can also drive the robot itself through the Jetson and the robot's controller, with a watchdog and speed caps. A campaign day runs a fixed protocol: calibration spins, a mapping drive, figure-8 laps at set speeds, ten-minute holds, back-and-forth motion to measure latency, cold starts, and outages where someone covers the sensor.

A VLA can look great near its training setup and fall apart a few centimeters away, so the hard part is deciding what a fair test is. We picked eight tasks across six kinds of challenge and changed the objects' placement and the wording on each of ten rollouts. We randomized the placements within a set workspace before running any evaluation, and every method ran the same ones. If I ran it again, the biggest thing I'd change is the lighting: we didn't control it well, and the models performed noticeably differently early in the morning than late at night.

Success alone also hides a lot, so in the main benchmark each rollout was scored three ways: did it succeed, what fraction of the goals were met, and how far along a written rubric it got. Separately from that benchmark, we ran an ablation on wording alone: one task with the apple held still and only the instruction changed, fifteen phrasings run once each. π0.5 succeeded on "Grab the apple and set it on the pan" and failed on "Take the apple and place it onto the pan." It was sensitive to how specific the prompt was: given an underspecified one like "Put the apple away," the arm wandered without targeting the apple before eventually turning to the task, and an overly specified prompt degraded it the same way. It struggles when the task's intent is missing or buried.

MolmoAct2 was strong on the banana but struggled with the other pick-and-place tasks, which suggests it's fragile to object type, and on the harder tasks it often misread the prompt entirely, pressing the top of the cabinet instead of closing the drawer. It also disengaged early: in the two-object task it moved one object and withdrew, and in the cluttered box it picked the wrong object and never came back for the right one. Across all of it, the VLAs failed in the same few ways: sensitivity to how the prompt was worded, dependence on the object (π0 never got the baseball into the bucket), and poor spatial reasoning. They ignored vertical clearance, like the height of a barrier or how high to lift a cloth before setting it aside, and when a task called for a specific procedure they defaulted to another one that couldn't finish it, grabbing the drawer's lip instead of pushing it closed.