As text-to-image models have improved in prompt adherence, the diversity of their outputs appears to have declined, a phenomenon commonly referred to as mode collapse. This benchmark evaluates output diversity holistically on the basis of large-scale human judgements, over a curated, deliberately non-restrictive set of 299 prompts spanning five tiers of specificity, from a bare subject (“a dog”) to open-ended requests (“something surprising”). For each prompt, every model generates a pool of images from which grids are sampled. Models are then compared in pairwise matches: annotators are shown one grid from each of two models and asked which exhibits greater intra-grid similarity, and the less similar grid is scored as the winner. The 200,100 votes cast across 40,020 such matches are aggregated into ratings with a Bradley–Terry model, such that a higher rating indicates greater diversity; the methodology details matchmaking and score estimation.
Table 1. Overall standings, Output diversity benchmark. Scores from a Bradley–Terry fit rescaled to Elo; intervals are 95% Wald intervals. Click any row to view sample outputs in Figure 1.
Figure 2. Overall Elo against the provider's list price, cost on a log scale. Teal-ringed models are the Pareto frontier, no model is both cheaper and better, joined by a step line, so picking a budget on the x-axis reads off the best model at or under it. Faded models are dominated; whiskers are the score 95% CI. Prices are hand-maintained snapshots of provider list prices. Hover a model for exact numbers.
We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.
