What is a good task?

Benchmarks lose trust for a reason. Most of the time, the model isn't the problem. It is a weak task, a grader that scores the wrong thing, or a path that hacks the reward without doing the work.

A task is a question, a way to score the answer, and the files or tools the system can use. At HUD we have helped shape tasks across long-horizon coding, robotics, browser work, and more, with many vendors.

What is a task?

Think of a task as an assignment a teacher gives a student. Three parts:

1. Prompt - the question you ask.
2. Success criteria and grader - what counts as done, and how you score it. Many tasks have more than one valid outcome. Sometimes you grade the reply. Sometimes you grade the resulting state, like whether a booking actually exists.
3. Environment - the files and tools available for solving it.

All three of these are not equally required, because sometimes you can leave out the environment and still have a clean question-and-answer task. But if you leave out the prompt, the whole task disappears. Same with success criteria; without them, you are reviewing vibes instead of results, which can be the case sometimes, but not here.

A lovely example: how many times does the letter R appear in “strawberry”? The prompt is the question, success is 3, and you do not need tools.

An environment is the world you give the agent for the task: the tools and files it can use, plus a small function that hands out the prompt and returns a score, and the agent harness stays outside. Here is that shape in code:

[Image blocked: A minimal HUD environment with one task template]

Calling count(sentence="strawberry", letter="r") mints one concrete task. You define the environment, send the prompt, and return a score. More detail: Creating an environment.

A running example

We judge tasks on three qualities, in order: realistic, fair, and hard. The rest of this article walks one booking request through each, because it looks like real work and it is easy to break in all three ways.

You are acting as a receptionist at a music school and a parent would like to book the first piano lesson for her daughter, Mia, for a full hour with Ms. Nowak. They will not be able to arrive before 15:30 and Mia has to be home for dinner, so the lesson must end by 18:00. It has to take place on a weekday in the week starting on Monday 10 August. Since Ms. Nowak only teaches on Mondays, Tuesdays and Thursdays, you should use the timetable to find out when she is free.

The booking request is the prompt. The environment holds the teaching schedule (and whatever tools the agent needs to read it). If you check the schedule, the right slot is 16:00 to 17:00 on Thursday, 13 August. That window and date are the success criteria the grader checks for.

You can check the whole trace from prompt to results here:

1. Realistic

The prompt should sound like something a real person would ask, and the work should be something someone actually does. The booking request above is a good example. A real receptionist gets messages like that.

The worst thing to do is to frame the prompt so it is obviously a benchmark, bolt on unrelated checks, or leak the rubric into the environment. Start from something real when you can: a request someone actually sent, a file or codebase someone really uses, a real UI, or a real mistake someone had to fix. Keep the whole analysis and scoring in the grader, not in files the agent can read.

A task can be graded perfectly and be truly difficult, but if nobody does this work in real life, there's no point measuring how well an agent does it for that job. Made-up tasks can still be useful for practice. They just say less about work performance when the ask itself would never show up on the job. Realism is also about the distribution of the work. If the real job is in Verilog, defaulting everything to Python already distorts what you are measuring.

Here are two versions of the booking task that aren't realistic:

In one version, the prompt is clearly AI-generated. Same schedule, same constraints, but dressed up in section titles and emojis:

# 🎯 Task: Optimal Piano Lesson Slot Identification
## 📋 Overview
Welcome! In this task, you will be evaluated on your ability to perform multi-constraint scheduling analysis. Let's dive in! 🚀
[…]
## ✅ Success Criteria
Your response will be considered successful if it:
- Correctly identifies the day (weight: 40%)
- Correctly identifies the time range (weight: 40%)
- Does not propose invalid alternatives (weight: 20%)
I hope this helps! Let me know if you have any questions. 😊

No receptionist has ever received this message. It even announces that this is an evaluation and shows the agent its own grading weights. The agent still finds Thursday and still scores 100%. Headings and emojis alone do not make a task invalid. The problem is that the ask no longer looks like real work.

Another example is combining two unrelated questions. One version keeps the booking prompt word for word, then ends like this:

…Use the schedule to work out when she's free. Also, what is 63 × 91 − 43 × 12?

Each part might work on its own, but together they just make a quiz, which isn't what real jobs look like.

Realism isn't just about the prompt. The files in the environment should look like real files, and nothing in the prompt or the files should make it obvious to the agent that this is a test. There also shouldn't be any shortcut that lets the agent score well without doing the work, and shortcuts can hide in places you didn't write.

One of our early tasks asked an agent to diagnose a failed build, and the agent scored 100% because the task bundle included a notes file with the correct diagnosis and the grading rubric, which it opened before it opened a single log. The answer was right, but the run told us nothing about whether it could diagnose a build, so we removed the file from the Docker image and read the next traces to see how the answer was reached. This is why we read the traces behind perfect scores as carefully as the failures. QA agents read a frozen snapshot of the whole attempt - prompt, steps, tool calls, logs, and reward - so you can see whether the miss came from the environment, the prompt, the trajectory, or a shaky grader, not just "the model failed."

The grading scale should also match what success means for that task. A mostly correct answer shouldn't get 10%, and a mostly wrong answer shouldn't get 90%. Partial credit is not always right, though. For a booking, getting the teacher and duration right but the date wrong may reasonably earn zero.

2. Fair

A task is fair if the grader gives the correct score to any answer the agent might give. No false positives and no false negatives.

Fairness breaks when the grader crashes on valid answers, relies on keywords, only accepts one exact phrase, or scores things the prompt never asked for. A better test: imagine giving the task to ten competent experts. Their answers will be worded ten different ways, and some may reach the result by a route you didn't expect. Each one should still get 100%. If your grader can’t handle that, it’s checking whether the answer matches yours, not whether it’s correct.

You can try a simple version of this test yourself before sharing the task. Write one answer you know is wrong and another that's right but worded differently from your reference answer, then check both with the grader. If the grader doesn't fail the first and pass the second, it still needs work.

A grader also has to handle solutions you did not anticipate. We learned this from a task where agents had to fix a bug in some attention code. Four agents produced working fixes, but the grader rejected all of them. Each agent stored a repeat count in a local variable, while the grader tested only the lines that used that variable and left out the line that defined it. The code therefore failed in the grader even though it worked as written. We fixed the grader to test the whole block, then confirmed that it accepted these solutions while still rejecting the original bug.

Another time the answer was right and the grader still said no. We asked an agent to go through air-quality sensor incidents and explain what had gone wrong in each one. The report it wrote identified all 38 incidents, put every one in the right category, and got the reason right for 37 of them. Then our automatic scorer (the grader) marked it down for fabricating evidence. The check itself was poorly defined. It treated any slash in the text as a citation to a file, so ug/m3, the unit for pollutant concentration, became a file path. The file didn't exist, so the missing file counted as an invented source. OR/WA triggered the same rule. A check meant to catch hallucinations was punishing ordinary language for an agent that had done exactly what we asked. So the model was fine, but the task obviously was not.

Here are four versions of the Ms. Nowak task, each with an unfair grader:

1. Vague prompt, strict grader: One version just says, "A parent wants to book a piano lesson, look at the schedule and let me know when might work." The grader still expects Thursday 16:00 to 17:00. The question doesn't give enough information for that answer, which is unfair.

2. Keyword "Thursday.": The grader marks any answer correct if it mentions Thursday. One agent replied, "Thursday is fully booked, I'm afraid. I can offer Wednesday 12 August at 16:00." Completely wrong, but it scores 100% because it includes the word Thursday. That's a false positive.

   

3. Exact wording: The grader only accepts the phrase "Thursday at 16:00 to 17:00." If an agent answers, "Ms. Nowak is free on Thursday the 13th from 4pm till 5pm," which is correct, it still scores 0%. That's a false negative.

   

4. Unprompted name criterion: One version splits the score in two: half for identifying the correct slot, and half for mentioning Ms. Nowak by name. The prompt never asked for her name. One agent found the right slot and wrote "she is free on Thursday at 16:00." Correct answer, 50%, because it said "she." Another agent said Ms. Nowak had no availability at all that week. Wrong about everything, also 50%, because the name was there. When a fully correct answer and a fully wrong one land on the same score, one of the criteria is broken. The fix is to delete it.

   

Reward hacking is not only a grading bug. A soft reward also teaches the model what to recommend next, including in places you never meant to score.

3. Hard

A task is hard when strong models struggle on it. We often use a bar like 50% or less, but that only means something under a named model, harness, scoring rule, and trial count. The number is a threshold you choose, not a definition of hardness, and one success in a handful of runs is noise. The real challenge is getting there without losing realism or fairness, so this check comes last in the order.

Making the prompt unclear, grading more strictly, giving one part of the answer too many points, or adding unrelated questions does not make the task harder. It just makes it less realistic. None of that makes the actual problem harder. Keep the request clear, and if current models already solve the task easily, find a different task instead of changing the grading.

At first, the only way to know is to run the task. After enough tasks, you start to notice which tasks are hard for current models, and your first drafts get better. That intuition is probably the most valuable thing a task creator builds, but it can become outdated quickly, because what was hard six months ago may not be hard now.

Real difficulty should come from the work required inside the environment. In a coding task, the answer might be spread across source files, configs, tests, and logs. In a browser task, it might mean navigating several pages, comparing information from different places, and keeping track of changes along the way. The environment can be messy and contain outdated or conflicting clues, just like real work, but the information needed to solve the task must still be there.

Hardness is most useful when it produces a recognizable kind of failure, not just a low average. Keep the ask open enough that a strong model can still solve it in more than one way.

For one task, we gave the agent the results of ten runs of a mixture-of-experts model, along with the code, configs, run records, and metrics. We left out our summary and asked the agent to work out what the experiments showed. At first, the runs looked successful because training loss, the score you minimize during training, improved every time. But the routing metrics, which show which experts actually handled the tokens, showed us that most tokens were going to only a few experts, so most of the mixture was unused. To answer correctly, the agent had to notice this conflict, separate the main experiments from the controls and irrelevant runs, and decide what the evidence actually supported.

From your work to a task

Somewhere in your work there is a real request, a concrete outcome, and some combination of files, websites, data, and tools needed to get from one to the other. That is a good place to start. Write what success means and how you will check it. Run it under a named model and setup so you know what “hard” means here, and read the traces, especially the ones that scored perfectly.

You can do all of this on HUD, against any frontier model. As earlier, QA agents can help you see problem points in the prompt or the environment, catch when an agent is reward-hacking the task, and flag when the task design itself looks fine.

Start from a template at hud.ai or build the environment yourself using the SDK docs.

If you already have raw workflows, logs, or datasets that should become tasks, that is useful too. We work with data vendors on turning existing enterprise material into training data, and with people who know the work well enough to shape new ones. Check it out on datavendor.ai and feel free to reach out if that sounds like you.