Reinforcement learning

In online RL an arm practises a task over and over, so after every try something has to put the peg back where the task starts. I built that reset for the peg-insertion task. A teammate and I designed the tagging together, and they wrote the code that reads it: ten AprilTags on the peg's faces, from which the scene camera solves the peg's full pose. I calibrated each camera with a ChArUco board, first its lens (the intrinsics), then its pose relative to the arm (the extrinsics). For the second, a calibration script has the arm set the tagged peg down at a grid of spots on the board and fits the arm's joint zeros and the camera's pose to where the arm thought the peg was against where the camera saw it. With that, the reset finds the peg anywhere in the workspace, works out how it is lying, and plans every waypoint of the pick and the place before the arm moves, choosing the grasp the arm can reach with the least travel within its joint limits. It then checks where the peg landed. This lets the teammate's automatic success detection score each try, so the arms can run online RL on their own. The reset has run on a real arm; the clip shows the same planner in simulation.

For V3, I wrote a full design document before writing any code. The target was concrete: simulate my club's six-robot arena of holonomic turret robots fast enough to train RL policies on a laptop CPU, since we don't have much GPU compute. Training takes millions of steps, so throughput was the goal, and V3 doubled as a performance learning exercise. One rule drove most decisions: only costs paid every frame matter. Only joint state, plus the contact solver's last solution, persists between frames. Everything else is either static data built at load time or scratch memory rebuilt each step and reused between passes. That rule became three pieces: fields; small operations that declare exactly which fields they read and write; and traversals that apply those operations across the joint tree, all checked at compile time. What the compiler can't check, the engine checks as it starts, and CI runs death tests confirming that a bad setup, like a model that doesn't fit its layout or an overflowing arena, stops it with a clear message instead of quietly corrupting a run. I kept V2's physically meaningful types but moved them into the math rather than the structure, so adding collisions or rendering meant adding passes instead of rewiring classes.