The Human-Preference Leaderboard for Output Diversity (Mode Collapse) Generation

Name: Output diversity benchmark v1
Creator: Rapidata
Published: 2026-10-29

Rapidatabenchmark@rapidata.ai

Abstract

As text-to-image models have improved in prompt adherence, the diversity of their outputs appears to have declined, a phenomenon commonly referred to as mode collapse. This benchmark evaluates output diversity holistically on the basis of large-scale human judgements, over a curated, deliberately non-restrictive set of 299 prompts spanning five tiers of specificity, from a bare subject (“a dog”) to open-ended requests (“something surprising”). For each prompt, every model generates a pool of images from which grids are sampled. Models are then compared in pairwise matches: annotators are shown one grid from each of two models and asked which exhibits greater intra-grid similarity, and the less similar grid is scored as the winner. The 200,100 votes cast across 40,020 such matches are aggregated into ratings with a Bradley–Terry model, such that a higher rating indicates greater diversity; the methodology details matchmaking and score estimation.

Leaderboard Compare Outputs Matchups Analysis Prompts Details

1. Motivation

Recent text-to-image models produce markedly better images than their predecessors, yet for an open prompt their outputs often look alike, an effect known as mode collapse that appears especially pronounced in newer models. Diversity matters wherever a model is used to explore ideas or to produce data, and for reinforcement-learning post-training, which learns from differences between sampled outputs (Liu et al., 2025). Albuquerque et al. (2025) likewise note that these models often produce homogeneous outputs, and call for robust diversity evaluation. One known contributor is classifier-free guidance, which trades diversity for sample quality (Ho and Salimans, 2022), particularly at high guidance scales (Sadat et al., 2024). This benchmark turns the apparent trend into a formal, comparative measurement: which models preserve the most variety when a prompt leaves room for it, as judged by people rather than by automated metrics.

2. Problem formulation

Attribute-based evaluations fix in advance which attributes should vary (Albuquerque et al., 2025), a design that keeps each judgement narrow. Our access to human judgement at a large scale lets us drop that restriction: we assess diversity holistically, with no predefined attributes, while the volume of votes keeps the resulting ratings consistent and reliable. Ideally, annotators would compare large collections of images from two models for the same prompt and judge which output distribution is more diverse.Since no annotator can take in hundreds of images at once, we compare small grids sampled from each model’s pool. Grid size trades overview against visibility: a larger grid shows more of a model’s range but leaves less attention for each image, so that similarity judgements may reduce to coarse cues such as colour scheme. We found 2×2 grids a good middle ground. Annotators see only the two grids, not the prompt, and are asked which looks more similar; the outcome is inverted so that the less similar grid wins; a smaller pilot study indicated that this phrasing gives more consistent judgements than asking which grid is more different.

3. Prompt set generation

The 299 prompts were generated for this benchmark with an LLM, from a written brief that set out what each part of the set should test, the intended share of every tag and rules for phrasing. The generated prompts were manually reviewed and the brief revised over several iterations. Every prompt has one tier, describing how much it constrains the image: a bare subject (T1, “a dog”), a subject in a setting (T2, “a lighthouse in a storm”), an event or situation (T3, “a wedding”), an abstract or emotional idea (T4, “nostalgia”) or an open-ended request (T5, “something surprising”). The tiers ensure that the set covers the full range of openness and allow results to be broken down by it; they do not prescribe what should vary, and annotators still judge diversity as a whole. T5 is kept small, as prompt adherence is barely defined there.Tags allow finer breakdowns. Concept type separates common subjects from rare ones, where collapse may stem from scarce training data, and fantastical ones, where it reflects a model’s own defaults. Cultural variability marks concepts whose appearance depends on region (“a wedding”, “breakfast”), and prompts with several subjects open up arrangement and interaction. No prompts were taken from public prompt sets, which models may have been tuned on. A tenth of the prompts are controls: style twins repeat a prompt with a named medium, and paraphrases reword one. Table 1 summarises the composition of the final set.

Table 1. Composition of the 299 prompts. Concept type and number of subjects apply to T1–T3 only; all other shares are of the full set.

Composition

Prompts per tier, T1 / T2 / T3 / T4 / T5

95 / 87 / 62 / 46 / 9

Concept type (T1–T3): common / rare / fantastical

62 / 15 / 23%

Multiple subjects (T1–T3)

17%

Culturally variable

24%

People

17%

Fixed medium

Style twins and paraphrase variants

10%

4. Image pool generation

Each of the 21 ranked models generated a pool of ten images per prompt through fal, at about 1024×1024, at its highest quality setting and with its default prompt rewriting. Every image has its own seed, sent explicitly wherever a model accepts one, as some models otherwise return identical images.Although the benchmark measures only diversity, it should not reward a catastrophic failure to follow the prompt: an image unrelated to the prompt inflates a grid’s apparent variety. Every T1 to T3 image therefore passes a deliberately lenient prompt-coverage check before grids are drawn. It passes even attempts that fail for what appears to be a lack of training data rather than disregard for the prompt, such as walruses drawn for “a narwhal” or eagles for “a griffin” (Figure 1). A broad first screen (Claude Haiku 5.5) flags anything doubtful, and a second judge (Claude Sonnet 5.5) rejects only images that are unrelated to the prompt or omit a subject it names; imperfect, partial or stylised attempts pass. T4 and T5 are not screened: their prompts are so open that almost any image is a valid answer, which makes a consistent automatic check infeasible. From the images that pass, we draw 2×2 grids of distinct images, no two with the same set.

Qwen-Image, “a narwhal” — Figure 1. Grids that pass the coverage check although the model evidently does not know the subject.

FLUX.1 Krea [dev], “a griffin” — Figure 1. Grids that pass the coverage check although the model evidently does not know the subject.

5. Discussion

While we have taken measures to ensure that models do not appear diverse simply by ignoring the prompt, such as using short, open prompts and screening out outright failures, we acknowledge and highlight that prompt adherence is intentionally not the focus of this benchmark. Its rankings should therefore ideally be read together with prompt-alignment results such as the Text-to-Image leaderboard.The prompt-coverage check described in Section 4 affected only 0.3% of model–prompt pairs and removed none of them entirely. Generation failures removed more. Five pairs are missing entirely because the model refused the prompt every time, among them both FLUX.2 pro and FLUX 3 Image on “a watermelon”; the others are Nano Banana on “a genie in a lamp shop” and GPT Image 2.0 and GPT Image 2.5 on “a superhero”. Those prompts do not count towards those models’ ratings.HunyuanImage 3.0 was excluded entirely after an initial run of the benchmark: on short prompts it frequently produced images that were unrelated to the prompt and often completely incoherent (Figure 2), a specific failure mode that would have inflated its apparent diversity. We hypothesise that these prompts are too short to sufficiently constrain its generation process, resulting in these artefacts.

HunyuanImage 3.0, “bread” — Figure 2. Grids from HunyuanImage 3.0 in the initial run, with images unrelated to the prompt or incoherent.

HunyuanImage 3.0, “a farmer” — Figure 2. Grids from HunyuanImage 3.0 in the initial run, with images unrelated to the prompt or incoherent.

Full protocol in the global methodology.

We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.

Benchmarks

Text-to-Image Output Diversity (Mode Collapse)Camera Movement Relative Camera Movement Physics Physics Realism Image-to-3D SVG LLM-to-3D Suggest a model or benchmark

Method

Methodology Data on Hugging Face

Rapidata

rapidata.ai Private benchmarks Contact X LinkedIn

Abstract