Models & Benchmarks

What does a benchmark score actually tell you?

That a model performed a specific task, on a specific distribution, under a specific prompting setup. Nothing more. A score is a measurement of a narrow slice of behaviour, and the gap between that slice and your workload is usually where disappointment comes from.

This matters because benchmark numbers are the primary currency of model launches, and they are almost always reported without the distribution they were measured on. “Strong at long-context retrieval” describes recall of a planted fact in low-noise filler. Synthesising across contradictory passages is a much harder task, and the phrase says nothing about it. Accuracy also varies by position within the window, which Liu et al. measured directly.

Why do vendor charts and your evals disagree?

Independent harnesses such as Stanford’s HELM exist because a vendor chart reports the configuration that flattered the model.

Usually because both are honest and measuring different things. Three recurring reasons:

• Task shape. Recall degrades slowly with context length; aggregation and absence-detection degrade quickly. A benchmark that measures the first tells you little about the second.
• Noise profile. Benchmark filler is unrelated text, so the right answer is the only plausible one in the window. Real corpora are full of near-duplicate passages that disagree.
• Prompt sensitivity. Reported figures often come from a tuned harness. The same model in your prompt scaffold is a different measurement.

None of this means benchmarks are useless. It means a score is a hypothesis about your workload rather than a conclusion. The only way to close the gap is to measure the thing you actually do.

How do you read a model card properly?

Read the evaluation methodology before the results table, and check three things: what was held out, what the baseline was, and whether the comparison models were evaluated under the same harness. If any of those is unstated, the numbers are directional at best.

Then look for what is absent. Capabilities a model card does not mention are usually the ones that did not improve, and that silence is frequently more informative than the metrics that are reported.

What should you measure yourself?

A small, fixed evaluation set built from your own traffic beats any public benchmark for deciding whether a model change helps you. It does not need to be large. Fifty well-chosen cases with known-correct answers will detect most regressions that matter.

Three properties make such a set useful. It must be stable, so results are comparable across months. It must include cases you currently fail, because a set you already pass measures nothing. And it must be scored the way the product is judged, which is rarely exact-match and frequently requires a human or a rubric.

The discipline that makes this work is boring: run it on every model change and every prompt change, record the result with the version that produced it, and never tune the set to make a release look good. An evaluation set that moves whenever the results are inconvenient is decoration.

How should you read this cluster?

Articles here work from primary sources, meaning model cards, papers and evaluation harnesses, and separate what a release measurably changes from what the launch post asserts. Where a number is quoted, the method behind it is described so you can decide whether it transfers to your case, and where something is a vendor claim it is attributed as one rather than restated as a finding.