Computer vision

In online RL an arm practises a task over and over, so after every try something has to put the peg back where the task starts. I built that reset for the peg-insertion task. A teammate and I designed the tagging together, and they wrote the code that reads it: ten AprilTags on the peg's faces, from which the scene camera solves the peg's full pose. I calibrated each camera with a ChArUco board, first its lens (the intrinsics), then its pose relative to the arm (the extrinsics). For the second, a calibration script has the arm set the tagged peg down at a grid of spots on the board and fits the arm's joint zeros and the camera's pose to where the arm thought the peg was against where the camera saw it. With that, the reset finds the peg anywhere in the workspace, works out how it is lying, and plans every waypoint of the pick and the place before the arm moves, choosing the grasp the arm can reach with the least travel within its joint limits. It then checks where the peg landed. This lets the teammate's automatic success detection score each try, so the arms can run online RL on their own. The reset has run on a real arm; the clip shows the same planner in simulation.

FTC rules ban streaming camera images to the driver station during a match, but telemetry text is allowed, so Live View sends the robot's camera feed as text. Each frame is shrunk and converted to the Lab colour space on the robot, then drawn with Braille characters as pixels: each character is a 2×4 block of dots, and Floyd–Steinberg dithering decides which dots are on, so shading survives with only on and off. To keep the message small, colour is added only for the game pieces: runs of characters that match the yellow, red or blue samples are wrapped in a colour tag, and only the 100 longest runs are kept. The text goes out as HTML telemetry ten times a second. It gave our drivers a view of what the robot could see during matches, within the rules.