/research
/research
Research

November 19, 2026

Yet Another Kernel Kernel (YAKK): Grading GPU kernels against adversarial computation

Yet Another Kernel Kernel (YAKK): Grading GPU kernels against adversarial computation

By Anish Sambamurthy

In this post, we describe how YAKK catches the cheating kernels which get past the correctness checks of widely used GPU kernel benchmarks. We used YAKK to identify 14 clear numerical errors in kernels written by METR's agents, to build a benchmark of 99 kernel optimization tasks from 39 functions, and to grade the kernels that 12 frontier models wrote for 10 of these tasks.

Key findings:

  • We wrote 21 families of cheating kernels. At least one got through each of the 10 widely used kernel benchmarks we tested, including KernelBench and GEAK, and none got past YAKK.
  • Among 208 agent-written kernels published by the AI safety group METR, YAKK found 14 clear numerical errors, all published as passing METR's own correctness check.
  • At KernelBench's current fp32 tolerance of 1e-4, an allclose check on one input passes 62% of the kernels YAKK rejects, and fails 40% of the ones YAKK accepts.
  • YAKK rejected approximately 1 in 8 of the kernels written by 12 frontier models.

Overview

YAKK, for "Yet Another Kernel Kernel," is a system for building GPU kernel optimization benchmarks that catch cheating kernels. If a function can be written and compiled in JAX, YAKK can turn it into a benchmark task for models to hill-climb against.

For each input, YAKK calculates the greatest error a legitimate kernel could produce at the precision the task author declared, and rejects any output outside that bound. YAKK's correctness checker also shifts inputs, then looks for appropriate changes in output response, to catch kernels that cheat by not reading all of their input.

TL;DR of our methodology:

  • We tested 12 models, listed in the appendix, at every reasoning setting each offers. Before submitting, models can run their kernel through YAKK and see its verdict 0, 1, or 5 times.
  • Our example tasks span nine families of Triton kernels on an H100, from typical LLM inference workloads to traditional numerical linear algebra.
  • Models are scored by speedup over compiled JAX[1], capped at the hardware roofline[2].
  • Kernels rejected by YAKK's correctness checkers score zero.

Why build GPU kernel benchmarks differently?

You can't walk fifty feet through a machine learning conference without tripping over half a dozen new benchmarks, and a big chunk of the recent crop is for "GPU kernels." What, then, is a GPU kernel? And why are we interested in creating benchmarks for them?

In mathematics and computing, "kernel" has historically had several meanings. At some point in the late '90s, researchers in stream processing (a style of computing that pushes a big stream of data through the same operation) adopted a specific definition: a function launched over every element of a parallel dataset. If you use a GPU to do this (to quickly run the same calculation over a huge amount of data), you have a GPU kernel. This process is fundamental to modern deep learning, so GPU kernels now do much of the computing behind today's AI models.

LLMs are increasingly being used to write GPU kernels, so it is becoming ever more important to be able to assess their performance quickly and accurately. Benchmarks do this by outputting a single number, or grade, which describes how good the kernel is.

Grading a kernel

A kernel is graded on two metrics: whether the kernel computes the correct function, and how quickly the computation is performed. In this post, we will focus on discussing the first, as it is most relevant to the gap that YAKK addresses. The two metrics are linked, however. The fastest way to compute a function on a GPU is rarely the obvious one. A fast kernel will rearrange its work to minimize memory usage while maximizing concurrent calculations. Mathematically, the answer should be the same regardless of the chosen method, but as computers round slightly at every step, doing the work in a different order changes the result a little. Any correctness check should allow for this variance.

Every one of the major kernel benchmarks we have surveyed checks the correctness of a test kernel in roughly the same way:

  1. Start with a trusted version of the function, known to give the right answer (the reference).

  2. Generate some random inputs.

  3. Run the inputs through the reference kernel and the test kernel, and check that every number the test kernel outputs is close to the outputs from the reference, using an allclose test:

In this kind of test, you have an error tolerance, split into an absolute part (atol) and a part relative to the size of the reference output (rtol). As discussed above, computers round numbers slightly at every step, so two perfectly correct kernels that do the same steps in a different order will result in slightly different answers. To account for this gap, the allclose test gives each output its own tolerance: the absolute part, plus the relative part times the size of that output's reference value.

In other words, you check the output of the kernel you are testing against the output of a trusted reference kernel, and if the outputs are all the same, give or take your allowance for error, the kernel passes the test.

There are two problems with this method of checking correctness. First, it collapses the outputs. Millions of output values are reduced to a single pass-fail test. We find out if all of the output values fall within the error budget, but we cannot further identify where they differed from the reference outputs, or if the outputs were systematically off in one direction.

The second issue here is less obvious. By doing this test, we have also collapsed our inputs. A kernel is supposed to work for any input it might be given. However, this test is only performed on a handful of random samples, with the assumption made that if the kernel works for those samples, it must work on all inputs.

These issues are non-issues when the kernel is honest and attempts to compute the right function. However, if LLMs are writing them, we cannot assume that we are grading honest kernels.

A market for lemons

A kernel task is in some way like a trade, with the human as a buyer and the LLM as the seller. When the seller knows more about the quality of what they are selling than the buyer does, bad products, or lemons, can crowd the market, unless the buyer has a method for checking the goods.[3] To avoid an influx of bad GPU kernels, we need a way for humans to robustly assess the kernels written by LLMs. As a practical matter (that we will elaborate on later in the piece), we would ideally like a way to do our evaluations while never needing to read the actual code.

So the LLM has a black box which it tells us will compute the function. As the buyer, we want to make sure the black box does what the LLM claims. How can we do this? We generate some inputs, run them through the black box, and then check the outputs. If they look as they should, we "buy" the kernel, and accept it is honest.

Humans are, presumably, incentivized to write kernels that do what they are supposed to do. With modern LLMs, however, if a model is trained to optimize for score, not correctness, it could well end up writing kernels that pass these checks without actually computing the correct function, whether deliberately or otherwise. The kernel could skip sections of the input, for example, or fail on large or awkward values, or the LLM could find a way to cheat the test. It becomes reasonable to assume its submissions are adversarial.

The makers of existing kernel benchmarks are aware of this, and expend a great deal of effort making sure that models don't cheat the grading process. Their threat model, however, is incomplete. The assumption typically goes that if the LLM was going to try and game the system, it would do so by breaking the grading process: a compile bomb, reading caches, stealing the answers… Yet, as we have already alluded to, these are not the only ways an agent can cheat. An LLM is often able to infer what sort of inputs we are likely to give it, and what the grading error will be. So instead of setting off a bomb, it can scam us with a lemon: compute a different, cheaper function that passes the numerical correctness tests it knows it will be graded on. The seller knows exactly how the car will be test-driven, so they know exactly which parts to stuff with sawdust.[4]

While existing benchmarks defend against adversarial code, we designed YAKK to address this kind of adversarial computation.

Is this a problem in practice?

Yes.

After reviewing established kernel benchmarks and examining their failures, we built 21 families of "lemon" kernels and tested them against 10 widely used kernel benchmarks. These kernels got through the correctness checks of every benchmark that we tested, including KernelBench and AMD's GEAK.

The crudest of these kernels shows the problem most clearly. One of the KernelBench tasks is a Frobenius normalization layer. This task involves taking an input tensor, squaring every element, summing them all, taking the square root, and dividing every element of the original tensor by that result.

KernelBench runs this calculation on a tensor of about 1.9 billion elements, filled with uniform random numbers between 0 and 1. Divide each of those by the square root of the sum of 1.9 billion squares, and the largest output you can get is about 0.00004. KernelBench's current tolerance for fp32 tasks lets each output be off by about 0.0001, more than twice the largest correct output. A kernel that returns all zeros is never off by more than 0.00004, so it passes. Earlier versions of KernelBench used a looser tolerance of 0.01, which an all-zeros kernel passes even more easily.

Passing all zeros is an obvious flaw, but it's reasonable to think that it could be fixed with a tightened tolerance. Unfortunately, it isn't. The sum under the square root, called a reduction, is the expensive part of the kernel. This sort of kernel does not do much computation relative to the amount of memory it must read, so its speed is set by how much data it has to use. These reductions force the kernel to read every element of the input tensor before a single output can be generated, often multiple times. The elements are numbers drawn from some distribution, in this case uniform random, so in expectation they are all identical. For a uniform distribution between 0 and 1, the expected squared value is 1/3. When summing many millions of elements, the sum sits close to N/3 in the same way a billion coin flips will have about half a billion heads.

So the norm is √(N/3), give or take one part in 100,000. A fast-looking lemon skips the reduction step, and just divides by the constant √(N/3). Remember, the agent gets a better grade if it works quickly. It reads the tensor once instead of twice, and it matches the reference closely enough that a tighter tolerance won't catch it, even though it would give the wrong answer for any input that isn't random noise.

The lemon kernels that we wrote to showcase these two problems pass KernelBench's own evaluator.

Benchmark authors are aware of these methods (which we call concentration attacks), and the natural first fix is to hide the input distribution by drawing a secret fill for every evaluation and warping each row by a random affine map, so there is no constant √(N/3) left to memorize.

These methods kill hardcoded constants, but they don't address the root cause of the agent's workaround. Using the constant value was effective because a sum over a billion random numbers is barely random at all, and hiding the distribution doesn't change that. In practice, regardless, many benchmarks don't redraw their inputs at all.

A dedicated cheater adapts. Instead of using a constant, why not compute the statistic from only some of the data, and estimate the rest? Read the first half of each row, and use it to guess the second. If you try spiking the fills with heavy tails, the cheater can approximate exactly how often they should do real computation, and estimate everything else. This pattern is well documented in adversarial machine learning. Even clever defenses tested against fixed attacks often fall to adaptive ones.[5]

Here are a few of the more trivial cheating lemon kernels we built that got through KernelBench's and GEAK's correctness checks, but did not pass through YAKK's.

Benchmark (tolerance)KernelThe shortcut
KernelBench (1e-4)Frobenius normReturn zeros / hardcode the normalizer
KernelBench (1e-4)MSE lossRead 4k of 1.1 billion inputs, return a formula
GEAK, ROCm tasks (1e-2)LayerNormSkip the mean/variance reduction on large shapes
GEAK, ROCm tasks (1e-2)SoftmaxHardcode the normalizing denominator
GEAK, ROCm tasks (1e-2)Flash attentionDelete the numerical-stability machinery
GEAK, TritonBench-G tasks (1e-3)l2_normCompute small shapes, hardcode rstd for N > 2,048

More broadly, our numerical cheats fall into two categories.

"Lemon" kernel typeBenchmarks passed
Concentration. Replace an expensive reduction, like a norm, a mean, or a softmax denominator, with a constant or a small sample, because on the benchmark's inputs it barely moves.KernelBench, GEAK, BackendBench, FastKernels, Atrex-Bench, KernelBenchX, MultiKernelBench
Structural. Change which inputs feed which outputs, let tokens acausally peek ahead, or skip terms too small to show up on the test inputs.GEAK, FlashInfer-Bench, BackendBench, Atrex-Bench, KernelBenchX, MultiKernelBench

There are other, more technical, variants in our family of lemons that go beyond the scope of this post. During testing, we discovered that exposure to these methods dramatically increased LLM attempts to cheat on our benchmark. We plan on writing a follow-up piece describing this behavior, along with a more detailed examination of frontier model traces when solving harder tasks on our benchmark.

How YAKK decides correctness

Lemon markets only exist because of an information gap, in this case between the LLM and the grader (which provides feedback to the human "buyer," to extend our metaphor). The LLM knows how it produced the outputs; the grader does not. The solution to the problems outlined above comes from targeting that gap.

We encounter a challenge here. If you want to catch a cheater on a math exam, you ask them to show how they solved the problem. GPU kernel optimization, however, is entirely about rearranging work order efficiently. We don't know which intermediate steps the model will take, so we can't always know what intermediate steps to check. On top of this, there are many layers of abstraction that can be opaque due to closed-source decisions from hardware manufacturers. To address this issue, we assume a setting where we cannot examine the source code at all, and can only look at inputs and outputs.

We might not be able to open the LLM's kernel, but we can open the reference, and it holds much more information than a naive check like allclose uses. Whoever writes a kernel benchmark task has a slow, naive reference implementation of the function, and that implementation is a computational graph: a calculation written out as a chain of small steps.[6] From those steps, we know how much rounding error each one can introduce, and which inputs feed into which outputs. allclose throws all of this away and keeps only the final answer.

Using computational graphs raises an obvious objection. The graph we have per task is only one graph, and there are infinitely many others that compute the same function. We can write f(x, y, z) = (x + y) + z as f(x, y, z) = x + (y + z), and both versions are just as correct. In ordinary math, they always give exactly the same answer. On a computer, they might not. Numbers are stored with limited precision and rounded after every step, so the order of steps matters.[7] We can't expect the LLM to use exactly the same steps as our reference, or perform them in the same order, especially as this will not result in a faster kernel. Knowing how much error our steps introduce doesn't tell us how much error theirs will.

So we flip the question. allclose holds the kernel as fixed, and asks whether its output is close to ours on a few random inputs. YAKK holds the input as fixed, and asks whether any "honest" kernel could have produced the output.

"Honest" needs a definition for each task, and the task author supplies it by declaring a precision. An honest fp32 kernel, for example, computes the right function in fp32 arithmetic, in whichever order it likes. From that definition, we walk through the computational graph[8] and ask: for this specific input, what is the worst error any honest kernel could produce?[9][10] Inside that bound, we don't care how the agent got there, using any algorithm, any instruction, and any order of steps. Outside of it, no honest kernel could have produced the answer, meaning the kernel cannot be honest. We call the evidence that the output is within the honest bounds a certificate.

Sometimes the rounding error compounds so fast through a calculation that the bound gets too loose to differentiate a clever kernel from a lemon. When that happens, YAKK abstains and falls back to empirical checks. This did not occur for any task in our benchmark sweep.

Holding the input as fixed does come at a price. One input receiving a certificate does not mean the next one will too: a kernel could be correct for one input, then cheat on another. YAKK therefore also uses the computation graph to select which inputs to test. The graph tells us which outputs depend on which inputs.[11][12][13][14] We take groups of these outputs that move together and collectively probe all their inputs simultaneously. This gives us a simple check that confirms the outputs have the correct structural response to the inputs, catching the lemons that ignore any part of their input, as well as subtler bugs like a causal attention kernel that peeks at future tokens. An all-zeros kernel doesn't respond to any change in its input, and for a kernel with a hardcoded normalizer, changing one input moves only one output, when every output should shift. Both of these issues therefore get caught by YAKK.

Deriving "honest," the error bound for each operation, is the expensive part. A major advantage of our approach is that we can do so without reading a single line of the LLM's code. A new task becomes merely a new reference graph, and the bounds and the probes come out of composing operations that have already been defined.

Testing YAKK on kernels we didn't write

YAKK rejected every one of the lemons we described earlier, but that's to be expected: we wrote those kernels deliberately, after identifying the problems with correctness checks, and we built YAKK to address those problems. The more interesting test is running YAKK on kernels we didn't write, which had already passed somebody else's grader.

The safety group METR published 222 KernelBench solutions written by AI agents, each checked by METR's own correctness test, with an absolute tolerance of 0.01. YAKK graded 208 of them. We left out the rest because they were either too large to run on the GPUs we used, or written for KernelBench problems outside of the set we tested.

YAKK rejected 49 kernels. We manually checked the source code for each of them. Of these, 14 had clear numerical issues, all published as passing METR's own check[15][16][17][18], and most of the rest were caused by undeclared precision shedding. It didn't reject any kernel known to be correct. The problem appears in independently written kernels, which YAKK can identify.

Our favorite is a convolution whose kernel body is pass # The detailed implementations for the computation would go here, published with a 9.53× speedup. It passed because PyTorch's memory allocator fills its empty output buffer with the reference's just-freed result. Run on its own, it returns 1/16 everywhere.

Our example tasks

Our 17 example functions cover nine families from attention and mixture-of-experts kernels seen in typical LLM inference workloads to traditional numerical linear algebra, such as the conjugate gradient method, where rounding error builds up over many steps. They come from a larger set of 39 functions that YAKK currently supports. They are meant to test LLM capabilities for typical inference tasks, and general knowledge of GPU architectures.

Each task gives the model a reference implementation of a function, written in JAX, along with the input sizes and the precision it should work in. The model's job is to write a faster version as a Triton kernel. We score the model on how much faster its kernel runs than the compiled reference, in the case that YAKK accepts it. A kernel YAKK rejects scores zero.

Many of these functions come in several variants, each a separate task, at different sizes, shapes, or precisions. The fastest kernels at a given shape often vary with size and precision. The type of fraudulent substitution also varies, so testing a variety here is important. Not every variant is listed below.

LLM inference

TaskFamilyWhat it computes
decode_gqaAttentionDecode attention with grouped-query heads, KV cache up to 32,768. Variants: batched, 16 KV heads
flash_causalAttentionCausal flash attention. Variants: larger batches, bf16
ragged_paddedAttentionCausal attention over padded sequences of different lengths
sliding_segmentAttentionCausal attention under a sliding-window and segment mask. Variant: head dimension 128
onehot_grouped_gemmMixture of expertsGrouped GEMM with one-hot routing. Variants: 16 to 32 experts; 32,768 tokens
swigluMLPSwiGLU block: gate, up, and down projections

ML building blocks

TaskFamilyWhat it computes
layer_bwdNormalizationLayerNorm backward, d = 8,192 to 32,768. Variant: d = 65,536
layernorm_forwardNormalizationLayerNorm forward
layernorm_gemm_prologueNormalizationLayerNorm fused into a GEMM prologue
log_softmax_longSoftmaxLog-softmax over long rows, up to 1,048,576 columns
dense_nchwConvolutionDense 2-D convolution, NCHW layout
int8_gemv_carrierQuantizationint8 GEMV with the weights encoded in fp32
int8_gemv_dequantQuantizationint8 GEMV with dequantization. Variants: 32 to 64 input vectors; n = 65,536

Numerics

TaskFamilyWhat it computes
block_lower_solveLinear algebraBlocked lower-triangular solve. Variant: n = 1,024
cg_laplacian3dLinear algebraEight conjugate-gradient iterations on a 3-D Laplacian, grids up to 384³
gram_syrkLinear algebraGram matrix XXᵀ. Variants: m = 2,048; m = 8,192; bf16 inputs with fp32 accumulation
sliding_mean_windowReductionSliding-window mean, window = 1/32 of the row. Variants: window = 1/8, 1/16, 1/64, and 1/128 of the row; rows of 65,536 columns and up

You can't tune your way out

After we ran the 12 models on the example tasks in our benchmark, we took the 1,777 final kernels from our sweep, across all three GPU-check settings, and ran allclose on one ordinary input at every tolerance from 1e-8 up to 10, and compared the answer against YAKK's verdict. We opt to trust YAKK here. On METR's kernels, where we checked every rejection by hand, it never rejected a correct kernel, and never rejected one without a real defect.

The 1,777 final kernels from our sweep (excluding kernels the model left unmodified), each checked with allclose on one ordinary input at every tolerance. YAKK accepted 1,557 and rejected 220. The dashed lines mark KernelBench's current fp32 tolerance (1e-4) and the 1e-2 tolerance used in KernelBench v0 and v0.1.

A false reject is a kernel YAKK accepts that allclose fails, and a false accept is a kernel YAKK rejects that allclose passes. If you tighten the tolerance, you stop letting cheats through at the cost of failing honest kernels that reorder sums. If you loosen the tolerance, you run into the opposite problem, and start accepting lemons.

To get a ballpark measure on how well allclose separates good and bad kernels, we add two error rates: the share of bad kernels it passes and the share of good kernels it fails. A perfect check scores 0%, and flipping a coin 100%. At its best tolerance, allclose scores 79%. At KernelBench's current fp32 setting of 1e-4, allclose passes 62% of the kernels that YAKK rejects, and fails 40% of the ones YAKK accepts. That makes an overall score of 102%. Any fixed tolerance is almost certainly too strict in some places, and too loose in others.

To summarize, you will not find an allclose threshold that gives you a reliable correctness check. The table below shows why YAKK rejected each kernel.

Of the 1,777 kernelsCountWhat the grade record shows
Accepted1,557Every output value certified within YAKK's bound on the inputs tested (about 7.7 trillion values); none fell back to an empirical check or abstained
Rejected220
Output values outside the allowed bound2081.5 billion out-of-bound values in total; 27 of the 208 also produced NaN or inf
Crashed4They also fail allclose at every tolerance
Passed the value check, failed another check8Input probe, identity, or determinism check

Yet Another Kernel Kernel: A Conclusion

For teams training models on kernel tasks, a scoring system based on allclose can reward kernels that are wrong, which risks teaching models to cheat. For benchmark builders, a fixed tolerance can't separate honest rounding error from cheating. The tolerance should instead come from the computation, as it does in YAKK.

As for the name, YAKK. Where did it come from? Loosely speaking, any correctness check is a pseudometric: a way of measuring distance to the reference that is blind to some differences. The set of kernels a check can't tell apart from the reference is its kernel, which is where our cheats live. Most of the work we have done has been on estimating this kernel and shrinking our bounds as tight as we can get them. Hence, yet another kernel kernel.

We want to cover more families of operators, and we want to widen our bounds to allow for better user control of algorithmic approximations. Right now YAKK measures (in a loose sense) the forward stability[19][9] of kernels. We are confident that a backward stability estimate can also be developed. We will publish a technical report with full details in the next few weeks.

Appendix: full results

We tested 12 models at all available reasoning settings. The first table shows each model's best result with 5 GPU checks per episode. Each task was run once, so a small difference between models is not guaranteed to be robust.

ModelLabReasoning settings testedBest mean speedup (5 GPU checks)
Opus 5.5Anthropiclow, medium, high, xhigh, max9.09× (max)
GPT-6 AstraOpenAIlow, medium, high, xhigh, max8.05× (xhigh)
Fable 5.1Anthropiclow, medium, high, xhigh, max7.77× (xhigh)
GPT-6.1 SolOpenAIlow, medium, high, xhigh, max7.36× (high)
Gemini 4 ArgonGooglelow, medium, high6.78× (medium)
Sonnet 5.5Anthropicoff, low, medium, high, xhigh, max6.63× (xhigh)
Muse WonderMetaminimal, low, medium, high, xhigh, max6.05× (high, max)
Kimi K3Moonshot AInone, minimal, low, medium, high, xhigh, max6.03× (xhigh)
Qwen 3.8 MaxAlibabanone, minimal, low, medium, high, xhigh, max5.64× (xhigh)
GLM-5.3Zhipu AIlow, high, max4.43× (high)
Gemini 3.1 ProGooglelow, medium, high3.06× (high)
Grok 4.7xAIminimal, low, medium, high, xhigh1.67× (minimal)

The second table shows mean speedup over compiled JAX across tasks with 5 GPU checks per episode, for every combination of model and reasoning setting we tested. "Rejected" is the percentage of that combination's final kernels, excluding kernels the model left unmodified, that YAKK rejected.

RankModelReasoning settingMean speedupRejected
1Opus 5.5max9.09×0%
2Opus 5.5high8.47×0%
3GPT-6 Astraxhigh8.05×0%
4Fable 5.1xhigh7.77×10%
5GPT-6 Astrahigh7.64×0%
6GPT-6.1 Solhigh7.36×0%
7GPT-6 Astramedium7.28×0%
8GPT-6.1 Solmax7.11×0%
9GPT-6 Astramax7.03×0%
10GPT-6 Astralow6.98×0%
11GPT-6.1 Solxhigh6.91×0%
12Opus 5.5xhigh6.88×10%
13Gemini 4 Argonmedium6.78×0%
14GPT-6.1 Solmedium6.76×0%
15Sonnet 5.5xhigh6.63×20%
16Gemini 4 Argonhigh6.29×0%
17Muse Wondermax6.05×0%
18Muse Wonderhigh6.05×0%
19Kimi K3xhigh6.03×0%
20Fable 5.1max5.97×20%
21GPT-6.1 Sollow5.84×0%
22Muse Wonderlow5.84×10%
23Gemini 4 Argonlow5.65×0%
24Qwen 3.8 Maxxhigh5.64×22%
25Fable 5.1high5.58×30%
26Fable 5.1low5.50×0%
27Qwen 3.8 Maxmedium5.47×11%
28Fable 5.1medium5.44×30%
29Muse Wondermedium5.26×20%
30Muse Wonderxhigh5.17×10%
31Sonnet 5.5high5.16×10%
32Qwen 3.8 Maxhigh5.12×10%
33Kimi K3high5.04×10%
34Opus 5.5medium4.83×30%
35Qwen 3.8 Maxlow4.67×10%
36Sonnet 5.5max4.54×20%
37Kimi K3max4.50×0%
38Kimi K3none4.50×20%
39Qwen 3.8 Maxmax4.43×0%
40GLM-5.3high4.43×30%
41Kimi K3medium4.34×10%
42GLM-5.3low4.11×30%
43GLM-5.3max3.80×40%
44Sonnet 5.5off3.39×10%
45Qwen 3.8 Maxminimal3.14×40%
46Opus 5.5low3.09×10%
47Gemini 3.1 Prohigh3.06×10%
48Sonnet 5.5medium2.88×10%
49Gemini 3.1 Promedium2.63×20%
50Sonnet 5.5low2.62×0%
51Muse Wonderminimal2.48×40%
52Kimi K3minimal2.47×30%
53Kimi K3low2.21×40%
54Grok 4.7minimal1.67×20%
55Grok 4.7medium1.63×30%
56Grok 4.7high1.47×30%
57Grok 4.7low0.77×20%
58Qwen 3.8 Maxnone0.65×20%
59Grok 4.7xhigh0.64×67%
60Gemini 3.1 Prolow0.55×30%
Cost per episode is the model's tokens at list price plus H100 check time, excluding grading.

References

  1. JAX documentation
  2. Roofline model (Wikipedia)
  3. George Akerlof, "The Market for 'Lemons'" (1970)
  4. Matilda (1996)
  5. Athalye, Carlini, and Wagner, "Obfuscated Gradients Give a False Sense of Security" (2018)
  6. JAX documentation, "Understanding jaxprs"
  7. David Goldberg, "What Every Computer Scientist Should Know About Floating-Point Arithmetic" (ACM Computing Surveys, 1991)
  8. Cousot and Cousot, abstract interpretation (1977)
  9. Nicholas Higham, Accuracy and Stability of Numerical Algorithms (2nd ed., SIAM, 2002)
  10. Solovyev et al., FPTaylor (2015)
  11. Curtis, Powell, and Reid (1974), estimating sparse Jacobian matrices
  12. Coleman and Moré (1983), estimating sparse Jacobians as graph coloring
  13. Gebremedhin, Manne, and Pothen, "What Color Is Your Jacobian?" (SIAM Review, 2005)
  14. Griewank and Walther, Evaluating Derivatives (2008)
  15. METR solution: a triangular mask kernel that runs on 0.02% of tiles but works because the inputs were already triangular
  16. METR solution: a batchnorm that skips both of its million-element reductions and uses its initial statistics instead
  17. METR solution: a kernel that has a 1 in 5 chance of passing allclose and apparently got lucky
  18. METR solution: one kernel with four separate errors that net out to under 1%
  19. Numerical stability (Wikipedia)
Read more research
OlderRECON: A benchmark for researching individualsOctober 21, 2026

Build environments that frontier labs trust

Create your first environment in minutes. No credit card required.

Anything you can simulate and grade, you can improve.

GitHubXDiscord
Product
  • Evaluations(opens in new tab)
  • Environments
  • DataVendor(opens in new tab)
Resources
  • Documentation(opens in new tab)
  • Blog
  • Research
  • Guides
Company
  • Careers39
  • Contact
© 2026 Human Union Data, Inc.
PrivacyTermsTrust(opens in new tab)