MethodologyAnnotator qualityUpdatesHugging Face436.7k downloadsSuggest a model or benchmarkRequest a private benchmark
Benchmarks
Dimensions
License
Models

Scores recompute with the real footage in the field.

Realism v1updated 2026-10-06

World Models Realism: AI vs. real camera footage

Rapidatabenchmark@rapidata.ai

Abstract

This benchmark ranks world and video generation models on whether their videos pass for real camera footage, judged blind against each other and against the real clip each scene was taken from. We took 25 real clips of everyday motion, from a plane landing to coffee being poured, and had every model generate four videos per clip from its opening frame and a one-line description. The rankings are updated continuously from 618,210 responses across 85,098 pairwise human-preference matches. All data and judgments are public; see the methodology for details on evaluation and scoring.

LeaderboardCompare OutputsMatchupsAnalysisPromptsDetails
1. Evaluation dimensions
Each prompt in the fixed set is evaluated separately across two dimensions. Each vote addresses only one dimension, using the dedicated question shown below.
Dimension
Question asked to annotators
Ranking
Photorealism
Which video looks more like it was shot by a real camera?
—
Visual Integrity
Which video has more glitches and is more likely to be AI generated?
InvertedThe output annotators pick loses the match.
Photorealism asks which of the two clips looks more like it was shot by a real camera. Visual integrity asks which has more glitches and is more likely to be AI generated, and is scored inversely, so the clip with fewer glitches wins. The real footage competes in both, blind, like any other entrant: its score is the bar a model has to reach to pass as real.
2. Prompt set version
The prompt set is 25 real video clips of everyday motion, held fixed to ensure reproducibility of the results. Each model received the opening frame of a clip and a one-line description of what happens in it, and generated four videos per clip, all filed under the clip’s identifier so that matches only ever pair videos of the same scene. The real clip itself is entered as one more participant, so annotators judge it blind alongside the generated videos. 25 clips is a small sample set; if there is interest, we can extend it with many more.
3. Model generation protocol
Every model generates from the same input (the opening frame of a real clip and a one-line description of what happens in it) using its standard available configuration. For each entrant, we record the provider, model identifier and generation date. Material changes to the model or generation configuration are evaluated separately.
4. Matchmaking
Matchups are sampled from the full pool, and each ranked model is included in at least 3,562 matches.
5. Judgements
This round aggregates 85,098 matches, each decided by the majority of at least 5 judgements from distinct annotators.
6. Annotator audience
The global audience: annotators qualify on validation tasks and are measured continuously by user score.
7. Public artifacts and limitations
The round is published as a Parquet dataset on Hugging Face: one row per matchup, carrying the prompt, both outputs, the weighted score of every dimension and each individual judgement with the annotator’s user score, country, language, gender and age. Not every entrant covers all 25 clips: Seedance 2.5 has 18, Alaya World, Cosmos 3 Nano and Yume 1.5 have 15, and Sana WM has 8. Rankings are human preference judgements from the global audience, with no automated metrics.

Full protocol in the global methodology.

We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.

Benchmarks
Text-to-ImageOutput Diversity (Mode Collapse)Multi-Step Image Editing (Identity Drift)Camera MovementRelative Camera MovementPhysicsPhysicsRealismImage-to-3DSVGLLM-to-3DLLM to AnimationSuggest a model or benchmark
Method
MethodologyData on Hugging Face
Rapidata
rapidata.ai Private benchmarksContactX LinkedIn
Rapidata, benchmark.ai © 2026