Data analysis

The question wasn't how accurate the Odin is but whether the sentry could rely on it alone, and one accuracy number can't answer that: the nav stack needs to know how the error behaves, whether it grows, and what happens when something goes wrong. So I designed the October 3 campaign around those questions (figure-8 laps, a long hold, repeated relocalizations and covered-sensor trials, all against mocap) and fit a small model to it: the true pose, plus an offset drawn each time the Odin relocalizes into its map, plus first-order Gauss-Markov noise: correlated over time, but decaying.

Splitting the error this way was the key decision. The offset from each relocalization turned out to be the largest term, which one overall noise figure would have blurred, and modelling the noise as decaying showed the error stays bounded instead of drifting. To check the model wasn't just describing the data it was fit to, I refit it without one session, simulated it many times along that session's real path (a Monte Carlo check), and asked whether the real error fell where the simulations said it would. It did: its real error there, 5.1 cm horizontal and under half a degree, matched the prediction on all six axes (roll came in just under its predicted range, still within the test). That made the recommendation clear: the Odin can be the sentry's only localization, kept on the turret major, where it swivels independently of the turret and the chassis. We tried it in several mount locations and found the occlusions made little difference. The nav stack only has to handle a rare wrong relocalization, and the output pausing while the sensor is covered.

I ran the same characterization on a WayPonDEV LD19 2D lidar, for obstacle mapping, with the Odin providing the pose. It scans at a steady 10 Hz with 8–14 ms of jitter, about 503 points per scan 0.72° apart, and skips 1–2% of scans. Range noise is 3–6 mm at 1–2 m, 6–11 mm at 2–3 m and 7–28 mm at 3–5 m: well under a 5 cm map cell, even at 5 m. At most 0.2% of returns are isolated false points and 1–2.4% are outliers, mostly at object edges where a beam half-hits the edge and half the background, which a filter removes easily. It never dropped off USB in the 40 minutes it ran, and the longest gap between scans was one skipped scan. The LD19 was good enough for our use, but we bought a 3D lidar anyway: the Odin's field of view is small, and a 3D lidar sees much more of the area around the robot.

Our robots know where they are from dead wheels: small unpowered tracking wheels with encoders that roll along the floor. When the engineer robot moved to new odometry pods with three of them, I rewrote the team's Kalman-filter odometry for them; I had taught myself Kalman filters from Kalman Filter from the Ground Up. The first version was one 8-state filter fusing the three wheels with the IMU. Its covariance between heading and the x and y motion turned out to be negligible, so I split it into three independent two-state filters, for x, y and heading, each updated by one measurement: every step now multiplies 2×2 matrices and divides by a number, instead of working through 8×8 matrices and inverting a 7×7 one, and the wheels alone turned out to be enough without the IMU. To tune it, I drove fixed paths (straight forward, straight sideways, spins and a square), logged the raw wheels, and replayed them offline to fit each pod's position, angle and wheel radius and the filter's noise, checking the result against the IMU's heading. Our target was under 10 cm of error over the engineer's path, the most our vision system could correct for, and the tuned odometry met it.

A VLA can look great near its training setup and fall apart a few centimeters away, so the hard part is deciding what a fair test is. We picked eight tasks across six kinds of challenge and changed the objects' placement and the wording on each of ten rollouts. We randomized the placements within a set workspace before running any evaluation, and every method ran the same ones. If I ran it again, the biggest thing I'd change is the lighting: we didn't control it well, and the models performed noticeably differently early in the morning than late at night.

Success alone also hides a lot, so in the main benchmark each rollout was scored three ways: did it succeed, what fraction of the goals were met, and how far along a written rubric it got. Separately from that benchmark, we ran an ablation on wording alone: one task with the apple held still and only the instruction changed, fifteen phrasings run once each. π0.5 succeeded on "Grab the apple and set it on the pan" and failed on "Take the apple and place it onto the pan." It was sensitive to how specific the prompt was: given an underspecified one like "Put the apple away," the arm wandered without targeting the apple before eventually turning to the task, and an overly specified prompt degraded it the same way. It struggles when the task's intent is missing or buried.

MolmoAct2 was strong on the banana but struggled with the other pick-and-place tasks, which suggests it's fragile to object type, and on the harder tasks it often misread the prompt entirely, pressing the top of the cabinet instead of closing the drawer. It also disengaged early: in the two-object task it moved one object and withdrew, and in the cluttered box it picked the wrong object and never came back for the right one. Across all of it, the VLAs failed in the same few ways: sensitivity to how the prompt was worded, dependence on the object (π0 never got the baseball into the bucket), and poor spatial reasoning. They ignored vertical clearance, like the height of a barrier or how high to lift a cloth before setting it aside, and when a task called for a specific procedure they defaulted to another one that couldn't finish it, grabbing the drawer's lip instead of pushing it closed.