When a new language model comes out, one of the most common ways to show it off is a three.js animation: a single clip of a scene the model wrote in code, chosen by the person posting it. This benchmark ranks language models on that task under fixed conditions. Every model receives the same 200 prompts, drawn from requests people post on X and type into chatbots and widened with scene descriptions from video benchmarks, and writes one HTML page per prompt that plays a three.js animation. Each page is recorded as a 7-second clip, and human annotators compare the clips of two models for the same prompt, without seeing the prompt or the code, and say which one they would continue watching and which one they like more. The 925,025 responses collected this way are aggregated into ratings with a Bradley–Terry model; the details describe how the prompts were chosen and the clips made, and the methodology covers matchmaking and score estimation.
Table 1. Overall standings, LLM to Animation benchmark. Scores from a Bradley–Terry fit rescaled to Elo; intervals are 95% Wald intervals. Click any row to view sample outputs in Figure 1.
Figure 2. Overall Elo of each model against its mean cost per sample, on a logarithmic axis. A sample’s cost is the sum of the charges the provider reported for the calls that produced it, and a model’s cost is the mean over its samples. Error bars are 95% confidence intervals of the score. Models ringed in teal and joined by the step line form the Pareto frontier: no other model has both a lower cost and a higher score. The remaining models are faded.
We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.