Continuously updated rankings of generative models from 3,688,094 votes across 895,268 pairwise human-preference matches. Quality is decomposed into capability dimensions: a fixed, curated prompt set paired with a single criterion, fitted with a Bradley–Terry model on the Elo scale. All data and judgments are public.
We evaluate generative model performance purely on human feedback. Humans perform inconsistently on absolute judgements, but much better in a comparative setting. Our rating is therefore based on pair-wise matches, the results of which are jointly fit using the Bradley–Terry model where the probability of model m beating model m′ is given by:
A match pairs outputs from two models for the same prompt. As evaluating all permutations is infeasible, matches are sampled, ensuring broad coverage per model. The resulting scores are scaled to the familiar Elo range via Rm = 400 βm + 1000 (2), which stretches standard Elo by ln 10 ≈ 2.3 to better highlight differences in model capability.
We curated a fixed set of prompts across 9 categories focusing on commercial applicability and ensuring reproducibility of the results.
To ensure trustworthy ratings we employ several measures, notably annotator pre-filtering via gold-standard questions, and redundancy, collecting a minimum of m responses per match (m = 3) and deciding the winner by majority. Read more
Figure A3. Each model's standing within each prompt category, scored on overall preference. Read a column: who is best in that category. Darkest is first. Hover or tap a cell to read its score, 95% interval and sample size; click a category to sort the rows by it. The number beside each model is its overall rank, so a jump reads as a jump. A cell reports a rank only above 500 matches for that model and is hatched below it, a rank on a handful of matchups is noise, not a result. How many prompts a category rests on is in its header and every tooltip: a narrow category asks a narrower question.
We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.
