This benchmark ranks generative models on whether they can continue a real physical event the way it actually happened, starting from the last frame before the event. The rankings are updated continuously from 1,532,948 votes across 306,574 pairwise human-preference matches. Quality is decomposed into capability dimensions: a fixed, curated prompt set paired with a single criterion, fitted with a Bradley–Terry model on the Elo scale. All data and judgments are public.
We evaluate generative model performance purely on human feedback. Humans perform inconsistently on absolute judgements, but much better in a comparative setting. Our rating is therefore based on pair-wise matches, the results of which are jointly fit using the Bradley–Terry model where the probability of model m beating model m′ is given by:
A match pairs outputs from two models for the same prompt. As evaluating all permutations is infeasible, matches are sampled, ensuring broad coverage per model. The resulting scores are scaled to the familiar Elo range via Rm = 400 βm + 1000 (2), which stretches standard Elo by ln 10 ≈ 2.3 to better highlight differences in model capability.
We took the 65 scenarios of Physics-IQ, one take and one perspective each, and had every model generate four videos per scenario from its scene description and starting frame, exactly as the Physics-IQ paper prescribes.
To ensure trustworthy ratings we employ several measures, notably annotator pre-filtering via gold-standard questions, and redundancy, collecting a minimum of m responses per match (m = 5) and deciding the winner by majority. Read more
Table 1. Overall standings, World Model Physics benchmark. Scores from a Bradley–Terry fit rescaled to Elo; intervals are 95% Wald intervals. Click any row to view sample outputs in Figure 1. The real footage is left out; scores are fit among the models alone.
Figure 2. Overall Elo against the provider's list price, cost on a log scale. Teal-ringed models are the Pareto frontier, no model is both cheaper and better, joined by a step line, so picking a budget on the x-axis reads off the best model at or under it. Faded models are dominated; whiskers are the score 95% CI. Prices are hand-maintained snapshots of provider list prices; 8 models without a recorded price are not shown. Hover a model for exact numbers.
We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.
