This benchmark evaluates the 3D understanding of language models: each model writes three.js code that builds an object from a short text prompt, and the object it builds is what gets judged. The rankings come from 135,152 pairwise matches, decided by 780,424 votes. Quality is evaluated in four dimensions: Alignment, Coherence, Creativity and Preference. All rankings are fitted with a Bradley–Terry model and reported on the Elo scale; see the notes for details.
We evaluate language models writing 3D code purely on human feedback. Humans perform inconsistently on absolute judgements, but much better in a comparative setting. Our rating is therefore based on pair-wise matches, the results of which are jointly fit using the Bradley–Terry model where the probability of model m beating model m′ is given by:
A match pairs the 3D objects two models built in code from the same text prompt. As evaluating all permutations is infeasible, matches are sampled, ensuring broad coverage per model. The resulting scores are scaled to the familiar Elo range via Rm = 400 βm + 1000 (2), which stretches standard Elo by ln 10 ≈ 2.3 to better highlight differences in model capability.
We wrote 500 text prompts in five categories of 100: vehicles, animals and creatures, buildings and architecture, furniture and household objects, and instruments and tools. All models share a ceiling of 64,000 output tokens per call, thinking included, and almost no call reaches it: every model's median call stays below 14,000 tokens (see the output and cost table).
To ensure trustworthy ratings we employ several measures, notably annotator pre-filtering via gold-standard questions, and redundancy, collecting a minimum of m responses per match (m = 5) and deciding the winner by majority. Read more
Figure A3. Each model's standing within each prompt category, scored on overall preference. Read a column: who is best in that category. Darkest is first. Hover or tap a cell to read its score, 95% interval and sample size; click a category to sort the rows by it. The number beside each model is its overall rank, so a jump reads as a jump. A cell reports a rank only above 500 matches for that model and is hatched below it, a rank on a handful of matchups is noise, not a result. How many prompts a category rests on is in its header and every tooltip: a narrow category asks a narrower question.
We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.
