Assemble Bench: Benchmarking Robot Models on Contact-Rich Assembly

Introduction

The problem

The best way to evaluate a robot policy is on real hardware in the real world. That's why robotics has a benchmarking problem: physical evals need expensive hardware and long setup times, the only way to run them concurrently is to buy and set up more robots, and reproducing a result means reproducing the entire physical environment, not just the embodiment.

One solution is to do evals in simulation. This solves setup time and reproducibility. Simulated evals, however, will always have to contend with the sim-to-real gap, which inhibits using the eval as a training signal.

Parallels with LLM training

A lot of people like to draw parallels between the training paradigm of LLMs and the coming training paradigm of generalist robotics models. Building on this parallel, we can roughly split robotics model training into three stages:

1. Pre-training: large-scale pre-training on messy data (internet videos, egocentric recordings, VQA);
2. Mid-training: smaller-scale action fine-tuning for a specific embodiment or task family;
3. Post-training: perhaps some more task-specific SFT and (crucially) reinforcement learning (RL).

True robotics progress will come from continual learning from experience – reinforcement learning – and this will require creating physical RL environments. HUD just so happens to be an RL environment company.

In short

We wanted to see whether we could produce a quality benchmark that – like benchmarks for coding agents – could also serve as a practical training environment, and in the process learn about friction points people typically experience when creating such environments.

Summary of physical environments

Characteristics of good physical environments

Let's take a first-principles look at the purpose of environments. Specifically we'll refer to environments for generalist robot models (whether real-life environments or digital simulated ones) as physical environments.

One way we think about the purpose of an environment at HUD is as a medium of learning.

This could mean, a human learning something:

1. evaluation: learning how well a model performs at some specific task
2. benchmarking: evals of individual models allow us to compare task performance across models

a model learning something:

3. training: a well-constructed environment can provide training signal for a model and enable its training through RL.

Many of the canonical benchmarking environments are currently only oriented toward the human learning side. While we are interested in applications 1 and 2 – i.e. evaluation and benchmarking – we're much more curious about application 3 – training.

Two gaps

The main reason current benchmarks can't be used for training is two interconnected "gaps".

First there is the "proxy gap". Any eval that doesn't involve the system actually performing the real task in the real world has some inherent proxy gap. Really all environments are proxies for "the real thing" and the proxy gap is how far your environment is from that real thing. For simulated environments the biggest component of that gap is the sim-to-real gap. A benchmark like RoboCasa365 ^[1] can serve as a proxy for how well a finetuned model can perform household tasks, but doing RL on a model in the sim won't lead to a robot zero-shot doing the dishes in a kitchen.

The second (and perhaps more insidious) gap is the "heuristic gap" – it's the difference between what you're trying to measure (by proxy) and what you actually end up measuring. A classic example of this in robotics is the LIBERO ^[2] benchmark. While it's often used as a heuristic for how well a model can perform diverse real-life tasks, it's been shown to actually evaluate rote memorization and not learned skill ^[3].

For a good training environment (and a good eval) the proxy and heuristic gaps are made sufficiently small such that a rollout in the environment can serve as a reliable training signal.

A quirk of robotics benchmarks

One quirk of most current robot model benchmarks is that they require fine-tuning the model on task-specific demonstrations, just to get any non-zero result.

This stems from the nature of the current generalist models themselves. We could simplify and say that publicly available models like pi0.5, MolmoACT, GR00T, etc. have only gone through the equivalent of stage 1 (pre-training) of the training paradigm discussed above. They are general pre-trained bases that require at least a stage 2 of fine-tuning just to be useful. Because of this required step for benchmarks, we should be especially careful of the proxy and heuristic gaps.

A standardized platform

One solution for the absence of stage 2 training would be having a large standardized SFT dataset with real-life data and a standardized robotics platform to evaluate a model. This way everyone can start from the same "stage 2".

This is what DROID ^[4] has become for the community. DROID refers both to a standardized robot platform (a 7DOF robot arm and a 1 DOF Robotiq gripper) as well as a large open sourced dataset ^[4]. Recent work like RoboLab ^[5] has shown that a DROID checkpoint trained on real-world data can be used to evaluate a policy without any additional training in a digital environment (real-to-sim). Crucially such standardized checkpoints can also be used to evaluate policies in real life – the most accurate and grounded form of evaluation ^[6].

Our simulated eval builds on this as we use the DROID platform ^[4]. The hope is that a model provider only needs to provide a checkpoint finetuned on the DROID dataset and that we can evaluate the model immediately (no user finetuning needed).

Although our benchmark can be used to evaluate DROID checkpoints zero-shot, in practice we found that (likely due to a custom frontal viewport for the external camera we used) even frontier DROID checkpoints like pi0.5 and Cosmos Nano-DROID achieved 0% across all tasks.

Therefore we generated a demonstration dataset of about 100 synthetic demonstrations and we used a pi0.5 DROID checkpoint ^[7] as our testing reference for the benchmark.

What we made – simulator, domain, and tasks

Our benchmark is based on the National Institute of Standards and Technology (NIST) assembly benchmarking procedure ^[8] and their Assembly Task Boards (ATBs).

These ATBs were designed in an effort to define standardized performance metrics and test methods for small-parts robotic assembly. Specifically, we use the ATB-1 taskboard and build on previous work like FORGE ^[9] and Factory ^[10].

The benchmark consists of 4 task suites and a total of 14 tasks with unique assets, as well as dense reward scoring for each of the task suites. We chose to start with the following 4 task suites (in rough order of difficulty):

These task families represent different subtasks and movements that may compose a full assembly process.

We built the benchmark on top of Isaac Lab Arena ^[11] – an early version of a compositional robotics benchmarking framework from NVIDIA. We chose it because we wanted our benchmark to minimize the sim-to-real gap as much as possible. Notably, Isaac Sim and its photorealistic rendering have been shown to enable real-to-sim evaluation ^[5].

The full suite of tasks is summarized in the table below:

Together with the benchmark we're also releasing the 1,355-episode demonstration dataset and a custom CG-DAgger training loop, both covered below.

Data generation

Diversity in the physical world

One thing we often fail to appreciate is how incredibly diverse the real world is. A coffee mug can come in a thousand different shapes and sizes, be of any color, weight, and material, as well as change its appearance just based on the lighting in an environment.

A generalist model should be able to deal with all of that.

Here we come to the topic of data. When constructing robotics datasets a key thing to consider is what kind of variation are you exposing the model to and what kind of variation you actually want the model to learn.

We can separate this variation into two types:

• competence variation – variation you want the model to learn (the behaviors, skills, lessons you want it to pick up on)
• perturbation variation – variation you want the model to be robust to and ignore (the color of a task table, a change in lighting, a slightly different camera view)

The model needs to train over such generalization dimensions ^[12] for useful performance. Real-life data produces such diversity naturally. Simulation has to create it.

Synthetic data

As mentioned before, to achieve non-zero success we needed to create a demonstration dataset for the benchmark. Since our tasks involve very fine alignment, generating this dataset through teleoperation in sim is difficult, and it leads to noisy, non-smooth motion from input devices.

Instead we used zero teleoperation in the creation of this data. Recent work has shown that coding agents can be useful for task-specific robotic control ^{[13] [14]}. Our full dataset – available on Hugging Face ^[15] – was generated entirely with such "scripted experts" that use privileged simulator state.

Of course synthetically generated data needs to possess some meaningful variation, and, as mentioned above, with synthetic data this variation needs to be manually encoded. We mostly focused on variation in demonstrated trajectories via per-episode randomization along a few small, task-meaningful axes (like grasp depth, lift height, etc.). Scene diversity (like object textures, scene background, and lighting) was largely ignored:

Data cleaning

One thing we learned first-hand during model training was that quality control (QC) for robotics data is very important. Prior to cleaning our dataset pi0.5 achieved 0% on all tasks.

We embedded each episode (resampled EE, state, and action plus a few scalars like length and grip stats), removed exact duplicates, then dropped:

1. Idle streaks: episodes with an idle streak of 15+ frames (our action chunk size)
2. Length outliers: episodes whose length was far from the per-task mean
3. Mahalanobis outliers: within-family outliers on the top PCA components – episodes that looked statistically unlike their peers. This also caught erratic, jittery motions that length alone missed.

We iterated this a few times and kept a handful of mild peg outliers that still looked like valid successes. PCA and t-SNE plots of these embeddings were surprisingly useful both for spotting outliers and for interpreting the data. The final published set totaled 1,355 episodes – ranging from 90 to 100 demonstrations per task.

RL

VLA models with diffusion action heads don't produce the action probabilities that most core RL algorithms need (PPO, DPO, GRPO, etc.). We tried model-agnostic methods like PA-RL ^[16] and PLD ^[17] instead (VLA-compatible RL deserves a blog post of its own), but they didn't train well on our tasks. This is likely for two reasons: fine alignment is brittle (a small exploratory move can destroy the reward signal), and the finetuned behavioral cloning (BC) checkpoint succeeded so rarely that there was little for RL to latch onto.

Therefore, we decided to stick to further BC and explored how the digital environment itself could be used to improve training. We limited the RL algorithm evaluation to one task suite – the round peg suite – and only trained the model on one task – the 8mm peg insertion – while the other tasks were used to evaluate generalization of the improved policy.

Specifically we implemented a variation of HG-DAgger ^[18] (Human-Gated DAgger; itself built on DAgger ^[19]). In this method the policy drives on its own, and a human only intervenes when things go wrong, taking over control and demonstrating a recovery. The policy is then trained on those intervention segments.

We keep the same loop, but drop the "human". Instead we detect when the model fails via code, and the "expert" that recovers is again a tweaked version of our scripted experts from the data generation step. We'll refer to this method here as Code-Gated DAgger (CG-DAgger).

We implemented 3 rounds of CG-DAgger:

Each training round we analyzed traces on the HUD platform to detect the key failure modes of a model, which then served as input to a coding agent to write the failure detection gate as well as the scripted expert to demonstrate the recovery behavior.

Example below: a scripted expert takeover during CG-DAgger collection on peg_round_8mm.

Results

Binary success on these tasks is sparse – often a handful of wins out of thirty – so it's bad for comparing checkpoints. We therefore report dense reward alongside sparse success, based on sparse milestones (lift, insert) plus continuous progress of movement toward next important locations, so partial credit accumulates even when the part never fully seats. All numbers below use the same protocol: 15 parallel envs × 2 waves = 30 episodes per task.

pi0.5 benchmark results

The BC baseline (pi0.5 SFT at step 12k) gets some traction on round pegs and nuts, and almost none on square pegs or small and medium gears. Nuts look surprisingly strong relative to gears – we did not randomize bolt pose, so the model may have learned a fixed alignment heuristic rather than real threading.

pi0.5 CG-DAgger results

Three CG-DAgger rounds on top of BC, trained only on peg_round_8mm corrections and evaluated across all round-peg sizes:

• After the first (grasp) round, mean dense reward went up while sparse success went down – the policy got closer more often, but seated less often.
• Later rounds improved the looser clearances (12mm and 16mm).
• The 4mm task stayed roughly flat; what we taught on 8mm did not transfer into that clearance.

Example below: a round 2 rollout exhibiting both recovery behavior learned during round 1 and a scripted expert takeover upon failure gate detection.

Future

For simulated RL environments in robotics to really take off, a few pieces are still needed. We hope to see better generalist models coming in the near future (that hopefully don't require task-specific SFT or at least come with quality DROID checkpoints).

In addition we hope to see better VLA-compatible RL algorithms that are cross-model compatible to enable simple task-agnostic RL for generalist robot models. Our current method, while successful, still requires human input. Someone has to review traces, recognize failure modes, and oversee the produced scripted trajectories.

Finally we hope to see simulators that keep closing the photorealism and physics gap. We are excited about emerging stacks like Isaac Sim 6 ^[20] with Newton and Genesis World ^[21].

Without these, evaluating and training robot policies in simulated environments still has a ways to go. Simulation – and synthetic data generation as a whole – is brittle. This goes back to the inherent diversity in the real world we discussed before. When creating synthetic data, every step is either explicitly or implicitly designed by a human. Think of the bolt placement in our simulation, the background scene we chose, or the arbitrarily chosen speed with which the synthetic demonstrations moved pegs.

"Designed" data comes with (often inadvertently) designed inductive bias ^[22]. Such biases push a model toward one specific way of sensing and acting (say one specific way of grasping a peg). In the real world, this impedes generalization, especially when many such biases stack up. To make simulated environments work, we need some way to effectively capture world-scale diversity and channel that into our environments ^[23]. We're excited to see where this endeavor goes.

We're open sourcing the benchmark and encourage contributions, as well as using the benchmark for research into evals and RL. The repository includes a HUD framework compatible environment that can be run locally and used to evaluate both VLAs and coding agents.

Repo · Dataset · Checkpoints