# Benchmarking VLAs for ARCHITECT

Human Centered Robotics Lab · Apr 2026 – present
Tools: Python, DROID, Franka Panda, openpi, GR00T, MolmoAct2, WebSocket, ZMQ
Page: https://ericjgoossen.com/projects/hil-cap/
Links: [GitHub](https://github.com/EricJGoossen/VLA_benchmarking) [Paper](https://arxiv.org/abs/2607.23784) [Website](https://robo-architect.github.io/)

Summary: The real-robot benchmark behind a paper on language-guided robot code, from the evaluation harness to what four VLAs can and can't do.
Lines of code Eric wrote: 2,031 (the sum of the entries below)

![A Franka Panda arm in a dark lab putting an apple and bread into a bowl, closing a drawer, and folding a cloth into a triangle](https://ericjgoossen.com/projects/hil-cap/images/hero-architect.jpg)

## Overview

In the Human Centered Robotics Lab, I built and ran the real-robot benchmark for ARCHITECT, the lab's system for writing robot programs with a coding agent and fixing them with plain-language corrections. Its paper compares that approach against vision-language-action (VLA) models, so I wrote the open-source harness that runs any VLA on a Franka Panda through DROID, designed the experiments, and benchmarked π0, π0.5, MolmoAct2 and GR00T N1.7 on eight real manipulation tasks, from pick and place to closing a drawer, folding a cloth and uncovering a banana hidden under one, plus an ablation of fifteen ways to word the same instruction. Along the way I learned where these models are strong, where they fall apart, and how to measure the difference fairly.

## Entries (3, newest first)

### The paper: A Few Words Go a Long Way (featured)

Date: 2026-07-26 · URL: https://ericjgoossen.com/projects/hil-cap/#paper-overview
Summary: The paper my benchmark fed: ARCHITECT, which fixes LLM-written robot code with a few words from a person, succeeded 70 to 100% of the time on all eight tasks.
Hardware: Franka Panda | Role: Co-author | Status: On arXiv
Skills: Robot learning
Links: [Paper](https://arxiv.org/abs/2607.23784) [Website](https://robo-architect.github.io/)

ARCHITECT is the lab's system, not mine: I'm a co-author for the VLA benchmark behind its comparison. The paper asks whether robot programs written by a coding agent, and fixed with a few words from a person, can beat end-to-end VLAs. ARCHITECT has an LLM write a program from perception and control tools, runs it, and reads back an execution trace. When it fails, a person says what went wrong ("grasp the cup from the top"), and the fix is saved to a skill library the agent reuses on later tasks.

On eight real tasks on a Franka Panda, **ARCHITECT succeeded 70 to 100% of the time on every one**, while Code as Policies, which writes the program once, topped out at 60%. The VLAs held their own only on simple pick and place, where π0.5 succeeded 90% of the time on the banana and every time on the block, against ARCHITECT's 80% and 70%; they scored 0 to 10% on the drawer, the cloth and the hidden banana. A six-person study tested the skill library: with skills learned on the cup task, people needed **0.83 corrections per trial on the box task instead of 4.67**, and with no corrections at all, success on the box task went from 0 of 6 to 4 of 6.

Video: Four rollouts in a two-by-two grid of moving a cloth off a banana and putting the banana on a plate: ARCHITECT, Code as Policies, π0 and π0.5
File: https://ericjgoossen.com/projects/hil-cap/images/compare-banana-under-cloth.mp4; still frame (an image you can view): https://ericjgoossen.com/projects/hil-cap/images/compare-banana-under-cloth.jpg
Caption: Banana under a cloth: ARCHITECT, Code as Policies, π0 and π0.5.

![Bar chart of human queries per trial with and without a skill library: about 4.7 without, under 1 with, one line per participant](https://ericjgoossen.com/projects/hil-cap/images/human-eval-queries.png)
Caption: Corrections needed, without and with skills from an earlier task (N = 6).

### An evaluation harness for any VLA (featured)

Date: 2026-07-10 · URL: https://ericjgoossen.com/projects/hil-cap/#harness
Summary: One open-source harness that runs any VLA on a Franka through DROID, whatever protocol its server speaks.
Lines of code Eric wrote: 1,853 | Language: Python | Tools: openpi, DROID, ZMQ | Hardware: Franka Panda | Role: Solo | Status: Open source
Skills: Python, Robot learning
Links: [Code](https://github.com/EricJGoossen/vla_benchmarking/tree/1ea375b)

Every VLA ships its own server, and they don't agree on how to talk: openpi's π0 and π0.5 use WebSockets, GR00T uses ZMQ and MolmoAct2 uses HTTP. I wrote [one harness](https://github.com/EricJGoossen/VLA_benchmarking) that hides all of that behind a single interface: a new model, task or whole benchmark is a short YAML file, not new code.

On the robot, the harness asks the policy for a chunk of actions, runs eight of them open loop, then asks again, recording the scene and wrist cameras throughout. The operator scores each rollout, everything is saved as it goes so an interrupted run resumes, and one script turns the results into success rates per task and per prompt.

![Diagram of the harness: YAML configs feed run_eval.py, which drives the DROID robot and asks one policy client for chunks of actions; the client reaches π0 and π0.5 over WebSocket, GR00T N1.7 over ZMQ and MolmoAct2 over HTTP; every rollout's scores and videos go to a results folder that compute_results.py turns into CSVs](https://ericjgoossen.com/projects/hil-cap/images/harness-diagram.svg)
Caption: How a run fits together.

Video: A Franka Panda arm running π0.5 picks up a banana and drops it on a white plate
File: https://ericjgoossen.com/projects/hil-cap/images/rollout-banana-plate-pi05.mp4; still frame (an image you can view): https://ericjgoossen.com/projects/hil-cap/images/rollout-banana-plate-pi05.jpg
Caption: π0.5 putting the banana on the plate.

### Designing the benchmark: what makes a VLA test fair (featured)

Date: 2026-05-15 · URL: https://ericjgoossen.com/projects/hil-cap/#experiment-design
Summary: How we made the VLA benchmark fair: eight tasks, randomized placements, three scores per rollout, and where the models failed.
Lines of code Eric wrote: 178 | Tools: YAML configs | Hardware: Franka Panda | Role: Team | Status: In the paper
Skills: Experiment design, Robot learning
Links: [Task configs](https://github.com/EricJGoossen/vla_benchmarking/tree/846182c/configs/task_configs)

A VLA can look great near its training setup and fall apart a few centimeters away, so the hard part is deciding what a fair test is. We picked eight tasks across six kinds of challenge and changed the objects' placement and the wording on each of ten rollouts. We randomized the placements within a set workspace before running any evaluation, and every method ran the same ones. If I ran it again, the biggest thing I'd change is the lighting: we didn't control it well, and the models performed noticeably differently early in the morning than late at night.

![Grid of success rates for eight tasks and six methods. The four VLAs score well only on simple pick and place, led by π0.5 at 0.9 and 1.0, and near zero on the drawer, the cloth fold and the banana under a cloth. Code as Policies peaks at 0.6. ARCHITECT scores 0.7 to 1.0 on every task](https://ericjgoossen.com/projects/hil-cap/images/sr-by-task.svg)
Caption: Success rate over 10 rollouts per task, from the paper's Tables 1 and 2.

![Filmstrips of the eight evaluation tasks on the Franka, each labeled with its challenge and instruction: three pick and place tasks, a two-object pick and place, picking bread from a cluttered box, closing a drawer, folding a cloth and uncovering a banana](https://ericjgoossen.com/projects/hil-cap/images/task-suite.jpg)
Caption: The eight tasks, from the paper's Figure 3.

Success alone also hides a lot, so in the main benchmark each rollout was scored three ways: did it succeed, what fraction of the goals were met, and how far along a written rubric it got. Separately from that benchmark, we ran an ablation on wording alone: one task with the apple held still and only the instruction changed, fifteen phrasings run once each. π0.5 succeeded on "Grab the apple and set it on the pan" and failed on "Take the apple and place it onto the pan." It was sensitive to how specific the prompt was: given an underspecified one like "Put the apple away," the arm wandered without targeting the apple before eventually turning to the task, and an overly specified prompt degraded it the same way. It struggles when the task's intent is missing or buried.

MolmoAct2 was strong on the banana but struggled with the other pick-and-place tasks, which suggests it's fragile to object type, and on the harder tasks it often misread the prompt entirely, pressing the top of the cabinet instead of closing the drawer. It also disengaged early: in the two-object task it moved one object and withdrew, and in the cluttered box it picked the wrong object and never came back for the right one. Across all of it, the VLAs failed in the same few ways: sensitivity to how the prompt was worded, dependence on the object (π0 never got the baseball into the bucket), and poor spatial reasoning. They ignored vertical clearance, like the height of a barrier or how high to lift a cloth before setting it aside, and when a task called for a specific procedure they defaulted to another one that couldn't finish it, grabbing the drawer's lip instead of pushing it closed.

![For π0 and π0.5 on each task, success rate beside task progression. On the hard tasks success is 0 to 0.1 while progression is often higher, such as π0 reaching 0.40 of the cloth fold](https://ericjgoossen.com/projects/hil-cap/images/success-vs-progress.svg)
Caption: Why one number isn't enough: success against how far the policy got.

![Fifteen phrasings of the same apple-to-pan instruction, with π0.5 and ARCHITECT marked for each. π0.5 succeeds on 6, failing on synonyms, reordered and ambiguous wording, typos and under- or over-specified instructions. ARCHITECT succeeds on 13](https://ericjgoossen.com/projects/hil-cap/images/language-ablation.svg)
Caption: The language ablation: one task, fifteen phrasings, each run once.