Evaluating Robot Models in the Cloud

Introduction

At HUD, we create RL environments to evaluate and train frontier AI models. Over the past few months, we've been working on utilizing the same infrastructure that enables highly parallelized and containerized evaluations of coding agents to evaluate frontier robotics models.

However, efficient robotics evals in the cloud is still quite greenfield, and we learned this the hard way.

Think of there being two sides to the eval process: the environment side, and the agent side. HUD lives on the environment side. We allow users to create and run evals on any CPU- or GPU-enabled robotics simulator they could think up. As for the agent side, people often take it for granted. With coding agents, we don't need to worry about hosting the model, batching its inputs for efficient GPU usage, and the latency that might result. We just query an API. Robotics models (until now) had no such analogue. And we had to build and provide that tooling ourselves.

What's more, robotics has a standardization problem. There's no accepted convention for interfaces and any eval requires at least a minimal "adapter" layer to interact with different embodiments and their input/output contracts.

So that's why we're excited to partner with Dreamscale. Their API and SDK enables us to evaluate robotics models in the cloud as efficiently as we evaluate coding agents.

In this blog post, we explore three different scenarios in which we use Dreamscale to evaluate different robot AI models in both simulation and the real world.

Experiment 1: Running VLAs at Scale

As robot developers post-train VLAs to improve performance, it's important to evaluate them across diverse tasks and environments in simulation to catch failure modes early before they are deployed to the real world.

Some of the common robotics simulation benchmarks have varying numbers of tasks, ranging from 60 tasks in RoboDojo ^[1] to over 10K tasks in LIBERO-Plus ^[2]. And if you are developing custom simulation environments, the number of task permutations (such as slight scene shifts) increases drastically. For any evaluation you'd then have to run hundreds or thousands of rollouts.

Therefore, a key challenge for sim evaluations is scaling up the inference compute to roll out hundreds of trajectories in parallel and reduce the time it takes to perform the evaluation.

We put Dreamscale's inference to the test. For MolmoAct2 ^[3] and π0.5 ^[4], we ran 128 simulated robots simultaneously against 128 different LIBERO ^[5] tasks. This could have been an arbitrary number from 1 to 128, as Dreamscale's scalable inference supports an arbitrary number of concurrent policies, with the underlying GPUs spinning up and down on demand.

Under the hood, Dreamscale leverages several tricks to achieve high inference throughput while keeping latency low.

• Disaggregation: splitting the VLA into the VLM module and action expert, and running them in parallel (maximizes throughput)
• Stream batching: the model allows requests to come in at any time without waiting for previous requests to fully complete (minimizes latency)
• Client-side orchestration: multiple clients coordinate when they should send fresh observations to the server (maximizes utilization)
• Urgency-based scheduling: embodiments that are closest to the end of their action buffer are prioritized (minimizes stalls)

HUD makes it easy to record the rollouts, view detailed traces, assign evaluation scores and review the action and observation history tick by tick.

Experiment 2: Running World Action Models in the Cloud

The physical AI field is waking up to a new trend in model architecture: World Action Models (WAMs) ^[6]. Instead of learning a direct map from vision to action, WAMs try to learn the causal dynamics of how the world progresses over time, making predictions about the future state and deriving the actions that lead to a desired state.

These WAMs are often built on top of video diffusion models to leverage the ability to predict future sequences of observations, which leads to better generalization across domains. However, because they must predict future state as well as future actions, WAMs are significantly more compute-intensive than VLAs, often exceeding the bounds of local compute (e.g. Jetson Orin and Thor).

One example is NVIDIA's Cosmos 3 ^[7], a multimodal model where a single Mixture-of-Transformers architecture jointly processes and generates language, image, video, audio, and action sequences. By passing in specific inputs (for our case, language instruction, observation video and proprioceptive state) and masking out the others, Cosmos 3 becomes a world action model that outputs action trajectories.

Cosmos 3 comes in three different sizes, from Edge 4B to Nano 16B to Super 64B, which gives a glimpse into the compounding scale of parameter counts we are seeing with newer world models. We take the Nano variant, which has a DROID ^[8] checkpoint available, and evaluate it on the RoboLab benchmark ^[9], a high-fidelity simulation benchmark released by NVIDIA in April 2026. It has a leaderboard that ranks over 15 submitted policies across a suite of 120 diverse tasks, where Cosmos 3 Nano Policy places 6th.

A recent addition to the RoboLab-120 leaderboard is FLUX 3 Action ^[10], a 7B world action model built by Black Forest Labs, based on their existing FLUX 3 vision backbone. It reached #1 upon release, demonstrating stronger performance with more efficient inference. However, it's still slower than the typical VLA, and only after a series of distillation and quantization steps does it start to match the inference speeds of VLAs.

Rather than trying to squeeze intelligence into the tight constraints of local compute, we envision a future where these larger, more capable models control robots over a real-time cloud inference API.

Again, we use Dreamscale's inference with HUD's environment framework to run FLUX 3 Action, demonstrating the ability to easily experiment with the latest world action models using flexible inference.

Experiment 3: Into the Real World

At HUD, we're beginning to extend our imagination beyond evaluations in simulated environments, towards physical rollouts on real robots. A key challenge here is that because frontier physical AI models have grown heavier in compute demand, you either need to pay up front for an expensive high-end GPU, or accept the slow inference loop as you watch your compute-constrained GPU struggle.

A core reason why we partnered with Dreamscale is their ability to serve inference of the strongest physical AI models over the cloud with real-time latency. Dreamscale applies a vast suite of optimizations across the full stack of inference: from the network to the GPU to the model architecture.

We ran a select number of real-world rollouts of π0.5 on a bimanual YAM robot setup via cloud inference, using HUD as the live trajectory recorder and evaluation tool.

Through a mixture of inference optimizations, network-level optimizations and real-time action chunking (RTC) ^[11], Dreamscale's cloud inference produced imperceptible delay (<50ms) and achieved smooth control at 30 Hz, quickly recovering from missed grips and slight bumps.

Conclusion & The Future

We're excited by what this partnership signals for robot developers in being able to use, evaluate and benchmark VLAs and WAMs, namely:

• Scalable cloud inference unlocking massively parallel sim evaluations
• Running powerful world action models that otherwise do not fit on conventional GPUs
• Bringing frontier physical intelligence to the real world via a frictionless API

Beyond evaluations, we envision a future where reinforcement learning drives the need for an even bigger number of rollouts being performed at scale, with HUD as the environment-side fabric on which Dreamscale provides agent-side inference. The possibilities are limitless: robots testing in and learning from an incredible diversity of real-world environments via cloud inference.

In collaboration with our friends at Dreamscale Labs, we are open-sourcing a ready-to-use cookbook where you can replicate the experiments and run cloud evaluations yourself.

Dreamscale x HUD cookbook on GitHub · Dreamscale