MethodologyAnnotator qualityUpdatesHugging Face416.9k downloadsSuggest a model or benchmarkRequest a private benchmark
Benchmarks
Dimensions
License
Models
Categories
Specificity tier
Content
Concept type
Subjects
People
Cultural variability
Medium
Headline set
Style
View
Output Diversity (Mode Collapse) v1updated 2026-10-29

The Human-Preference Leaderboard for T2I Output Diversity (Mode Collapse)

Rapidatabenchmark@rapidata.ai

Abstract

As text-to-image models have improved in prompt adherence, the diversity of their outputs appears to have declined, a phenomenon commonly referred to as mode collapse. This benchmark evaluates output diversity holistically on the basis of large-scale human judgements, over a curated, deliberately non-restrictive set of 299 prompts spanning five tiers of specificity, from a bare subject (“a dog”) to open-ended requests (“something surprising”). For each prompt, every model generates a pool of images from which grids are sampled. Models are then compared in pairwise matches: annotators are shown one grid from each of two models and asked which exhibits greater intra-grid similarity, and the less similar grid is scored as the winner. The 251,795 votes cast across 50,359 such matches are aggregated into ratings with a Bradley–Terry model, such that a higher rating indicates greater diversity; the methodology details matchmaking and score estimation.

LeaderboardCompare OutputsMatchupsAnalysisPromptsDetails

We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.

Benchmarks
Text-to-ImageOutput Diversity (Mode Collapse)Multi-Step Image Editing (Identity Drift)Camera MovementRelative Camera MovementPhysicsPhysicsRealismImage-to-3DSVGLLM-to-3DSuggest a model or benchmark
Method
MethodologyData on Hugging Face
Rapidata
rapidata.ai Private benchmarksContactX LinkedIn
Rapidata, benchmark.ai © 2026