As text-to-image models have improved in prompt adherence, the diversity of their outputs appears to have declined, a phenomenon commonly referred to as mode collapse. This benchmark evaluates output diversity holistically on the basis of large-scale human judgements, over a curated, deliberately non-restrictive set of 299 prompts spanning five tiers of specificity, from a bare subject (“a dog”) to open-ended requests (“something surprising”). For each prompt, every model generates a pool of images from which grids are sampled. Models are then compared in pairwise matches: annotators are shown one grid from each of two models and asked which exhibits greater intra-grid similarity, and the less similar grid is scored as the winner. The 200,100 votes cast across 40,020 such matches are aggregated into ratings with a Bradley–Terry model, such that a higher rating indicates greater diversity; the methodology details matchmaking and score estimation.
Figure A2. Each model's standing within each prompt category, scored on overall preference. Read a column: who is best in that category. Darkest is first. Hover or tap a cell to read its score, 95% interval and sample size; click a category to sort the rows by it. The number beside each model is its overall rank, so a jump reads as a jump. A cell reports a rank only above 500 matches for that model and is hatched below it, a rank on a handful of matchups is noise, not a result. How many prompts a category rests on is in its header and every tooltip: a narrow category asks a narrower question.
We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.
