Back to Resources
•Guide

Train an LLM to Play a Game with GRPO in ~100 Lines (2048 example with HUD)

A technical guide to the HUD RL-training cookbook.

The fastest way to understand GRPO is to watch a model get better at something you can see. So: take 2048, wrap it as an environment, and train a small model to play it. The whole thing is about 100 lines. The RL parts (environment contract and training loop) are closer to twenty-five. No Docker, no observation_space, no vLLM cluster. Runs on your laptop, scales to remote boxes without changing a line.

The full code is in the cookbook. This guide walks through why it's shaped the way it is.

Why a game is the best way to learn GRPO

GRPO needs a task the model is right about sometimes — not always, not never. 2048 delivers that naturally. The reward is the highest tile reached, the model lands all over the distribution, and you can watch the number go up and immediately believe it.

GRPO's trick is skipping the value network. Instead of a critic, it samples a group of attempts at the same task and scores each one relative to the group's average. Beat the mean and get a positive advantage. Fall below it and the advantage is negative. The group is its own baseline, and natural game variance does the work a critic would otherwise have to.

The environment, in ~40 lines

A HUD environment is a Python generator with two yields. The first sends the prompt to the agent. The second scores the result. Everything in between is just the agent working, with no observation_space, action_space, or float-array translation layer.

Here's the contract for 2048. The game logic (the Game2048 class that merges tiles and spawns new ones) is ordinary Python. What makes it an environment is the tool and the template.

# game2048_env.py (trimmed to the environment contract)

game = Game2048()
server = FastMCP(name="game2048")

@server.tool
def move(direction: str) -> str:
    """Slide the board: up, down, left, or right. Returns the board."""
    d = direction.strip().lower()
    if d not in _MOVES:
        return f"invalid direction {direction!r}; use one of up/down/left/right"
    changed = game.move(d)
    note = "" if changed else " (no tiles moved — try another direction)"
    over = "\nGAME OVER" if game.game_over() else ""
    return f"{game.render()}{note}{over}"

env = Environment(name="game2048")

@env.template()
async def play(target: int = 256):
    """Play one game; reward scales with the highest tile reached (target = win)."""
    game.reset()
    yield (
        "You are playing 2048 on a 4x4 grid. Each turn call the `move` tool with a "
        "direction (up/down/left/right) to slide and merge tiles. Keep playing to "
        f"build the largest tile you can (aim for {target}). The current board:\n\n"
        f"{game.render()}"
    )

    max_tile = game.max_tile()
    # Reward: normalized log2 progress from the start tile (2) to the target.
    denom = math.log2(target) - 1
    reward = 1.0 if denom <= 0 else (math.log2(max_tile) - 1) / denom
    yield EvaluationResult(
        reward=max(0.0, min(1.0, reward)),
        content=str(max_tile),
        info={"max_tile": max_tile, "score": game.score, "target": target},
    )

Three things to notice:

  • Observation and action are plain text. No space to satisfy, no encoder to write.
  • The grader reads game state, not words. Reward is a normalized log-scale of tile progress, smooth as the policy improves.
  • The move tool is a normal Python function. It is served over MCP but written like any other function.

Because the agent drives its own multi-turn loop, each move is a separate turn and a separate trainable unit.

The training loop, in ~5 lines

First, fork a model so you own weights you can advance in place. You can't train a shared catalog model — it will 404.

hud models fork Qwen/Qwen3.5-4B --name game2048-rl

Prereq note: HUD's SDK supports Python >=3.11 and <3.13. Python 3.12 is the safe default; uv run keeps that environment pinned so you don't accidentally install into 3.13.

Then the loop:

taskset, runtime = load_taskset_and_runtime()
session = await Job.start("game2048-rl", group=8)
for step in range(steps):
    start = len(session.runs)
    await taskset.run(agent, runtime=runtime, job=session)
    batch = session.runs[start:]
    await trainer.step(batch, learning_rate=1e-5, group_size=8)

The loop is three steps:

  1. Roll out. taskset.run plays one task, replayed into 8 independent games by Job(group=8) — one GRPO group.
  2. Collect. Each finished game is a Run with its full trajectory and reward.
  3. Train. trainer.step computes group-relative advantages, runs one gradient step, and promotes the new weights — all server-side. MODEL always points at the current policy.

Where the multi-turn magic actually lives

A 2048 game is a dozen-plus moves, but GRPO's reward is one number for the whole game. Credit still flows to each move because HUD records every turn as its own trainable sample. With return_token_ids on, each tool call carries token ids and per-token logprobs. HUD assembles them into a variable-length trajectory, broadcasts the episode reward across all turns, and the group baseline turns it into a per-token advantage. A 9-move game and a 31-move game are both just trajectories.

You can prove this to yourself before training anything. The cookbook's play_2048.py script runs a single game and reports how many turns carried trainable token-level samples:

$ uv run play_2048.py --target 256 --max-steps 30
playing one game (target=256, max_steps=30)...
reward=0.583 status=completed
agent turns=18 (with tool calls=18) trainable turns=18 tokens=4213
final: EvaluationResult(reward=0.583, content='64', info={'max_tile': 64, 'score': 932, 'target': 256})

(Illustrative output. Run the cookbook to see your actual numbers.)

This examples uses 18 moves for 18 trainable turns.

Watching it learn

The loop above prints the mean reward for each optimization step. A healthy run eventually looks like reward climbing while the within-group spread stays alive, though the first few steps on a small 4B model can be much flatter:

step  0 | reward 0.198 best_tile   32 | rollout  11.2s   2984tok  266tok/s | train   1.6s loss +0.0512 | optim 1 datums 88 failed 0/8
step  1 | reward 0.221 best_tile   32 | rollout  10.9s   3021tok  277tok/s | train   1.7s loss +0.0468 | optim 2 datums 91 failed 0/8
step  2 | reward 0.276 best_tile   64 | rollout  11.5s   3187tok  277tok/s | train   1.7s loss +0.0393 | optim 3 datums 94 failed 0/8
...
step 14 | reward 0.512 best_tile  128 | rollout  12.1s   3402tok  281tok/s | train   1.8s loss +0.0201 | optim 15 datums 102 failed 0/8

Two numbers to watch:

  • mean_reward trending up. The policy is improving.
  • reward_std staying non-zero.There's still within-group variance for GRPO to exploit. If the spread collapses to zero, the policy has saturated the task and the gradient dries up. That's your signal to raise the target tile or move to a harder game.

If reward is flat at 0.0 or 1.0 from step 0, the task is mis-scaled. Pick a target the base model hits sometimes.

Local first, then scale

Everything above runs on your laptop. When you outgrow it, you change one line:

# Local: rollouts run as subprocesses on your machine
runtime = LocalRuntime("game2048_env.py")

# Remote: rollouts run on leased HUD boxes, next to the env
runtime = HUDRuntime()

The training loop doesn't change. trainer.step only ever touches job.runs — it doesn't care where the trajectory came from.

Adapting it to your own task

2048 is a teaching example, but its shape carries over to real tasks. To train on something real, you change three things and nothing else:

  • The game logic becomes your task's logic, such as a checkout flow, a spreadsheet, a codebase, or a support ticket.
  • The tool becomes the actions your agent can take, such as click, edit_cell, run_tests, and send_reply.
  • The reward becomes what "good" means, such as task completed, tests pass, or correct answer.

The template and training loop don't change at all. In HUD's Sentry case study, this same loop trained a subagent from 6.3% to 13% success on hard tasks in about 13 hours. What separates a toy game from a production agent is the environment you write. The RL machinery is identical.

→ Clone the cookbook and run it

FAQ

Do I need my own GPUs?

No. Rollouts run locally as subprocesses. The training runs on a HUD gateway model through TrainingClient. (If you want to own the whole inference-and-training stack on your own GPUs, that's the verifiers / prime-rl approach instead.)

Which models can I train?

Trainable gateway models. Run hud models and look for the ones marked trainable (the example forks Qwen/Qwen3.5-4B, shown in the catalog as Qwen3.5 4B on a Tinker backend). Start small; a 4B model learns 2048 fine and keeps rollouts cheap.

Why GRPO instead of PPO here?

GRPO drops the value network and uses the group average as its baseline, which is exactly what a high-variance, verifiable-reward task like a game wants. HUD also supports RFT training for other model backends; run hud models to see what's available for your use case.

Is this single-turn or multi-turn?

Multi-turn. Each move is one agent turn with its own trainable token-level sample, and the episode reward is assigned across all of them.

Can I see the trajectory before I train?

Yes. Run play_2048.py to play one game and print how many turns carried trainable samples. It's the cheapest way to confirm your env produces the data the loop needs.