Our benchmarks evaluate generative model performance purely on human feedback. We elaborate the framework below. Each round additionally publishes a benchmark protocol recording its specific parameters.
1. Scoring
Humans perform poorly and inconsistently on absolute pointwise judgements, but are much better at comparative judgements [1]. To rank and rate models we therefore compute a rating from a series of pairwise matches [2]. Final scores come from a Bradley–Terry model, which models the probability of model m beating model m′:
The Bradley–Terry scores are scaled to match the familiar Elo range:
An important property of the Bradley–Terry model over a sequential Elo scheme is that it jointly optimizes over all matches. It is therefore insensitive to match order, and in turn more stable. Confidence intervals are computed by bootstrapping: resampling from the judgements with replacement and reporting the 95% interval.
2. Matchmaking
A match is a pairing of generated outputs from two models based on the same input. Evaluating all permutations is infeasible, so matches are sampled from the pool without replacement. Unlike user-driven arenas, the same output can be reused across matches against different opponents, which increases fairness as all models are evaluated on identical inputs. For major benchmarks we ensure at least 10,000 matches per participating model.
3. Prompts
We use prompts generally for any model input. Prompts are manually curated to be relevant to the evaluation criteria and to give good coverage of meaningful use cases, favoring commercial applications over viral trends. To balance coverage and tractability we curate roughly 1,500 prompts for major benchmarks.
Initial general benchmarks use prompts written in English; this is without loss of generality, as translating a prompt into English can be handled by an additional model step. Since annotators may not read English, they are shown a translation based on their device language and can toggle to the original.
4. Annotator reliability
Avoiding noisy or low-effort responses entirely is not possible on any crowdsourcing platform, so we employ several measures. Annotators qualify on validation tasks with known answers before their judgements count, and each match collects multiple judgements (by default five) with the winner decided by majority. A reliability estimator combines validation performance and continuous signals into a per-annotator user score used to weight judgements.
5. Capability dimensions
Most arena-style frameworks measure only end-to-end preference. In a research setting one usually wants more granular insight into where a model is strong or weak. A dimension is the combination of a prompt set and a single criterion on which annotators choose. We hypothesize that carefully constructed dimensions represent atomic capabilities that can be studied individually or recombined with weights. The end-to-end signal produced by other frameworks can then be approximated by the right combination of dimensions, which we examine in a companion analysis.