This benchmark evaluates the 3D understanding of language models: each model writes three.js code that builds an object from a short text prompt, and the object it builds is what gets judged. The rankings come from 135,152 pairwise matches, decided by 780,424 votes. Quality is evaluated in four dimensions: Alignment, Coherence, Creativity and Preference. All rankings are fitted with a Bradley–Terry model and reported on the Elo scale; see the notes for details.
We evaluate language models writing 3D code purely on human feedback. Humans perform inconsistently on absolute judgements, but much better in a comparative setting. Our rating is therefore based on pair-wise matches, the results of which are jointly fit using the Bradley–Terry model where the probability of model m beating model m′ is given by:
A match pairs the 3D objects two models built in code from the same text prompt. As evaluating all permutations is infeasible, matches are sampled, ensuring broad coverage per model. The resulting scores are scaled to the familiar Elo range via Rm = 400 βm + 1000 (2), which stretches standard Elo by ln 10 ≈ 2.3 to better highlight differences in model capability.
We wrote 500 text prompts in five categories of 100: vehicles, animals and creatures, buildings and architecture, furniture and household objects, and instruments and tools. All models share a ceiling of 64,000 output tokens per call, thinking included, and almost no call reaches it: every model's median call stays below 14,000 tokens (see the output and cost table).
To ensure trustworthy ratings we employ several measures, notably annotator pre-filtering via gold-standard questions, and redundancy, collecting a minimum of m responses per match (m = 5) and deciding the winner by majority. Read more
Table 1. Overall standings, Text-to-3D with LLM code benchmark. Scores from a Bradley–Terry fit rescaled to Elo; intervals are 95% Wald intervals. Click any row to view sample outputs in Figure 1.
Figure 2. Overall Elo against the provider's list price, cost on a log scale. Teal-ringed models are the Pareto frontier, no model is both cheaper and better, joined by a step line, so picking a budget on the x-axis reads off the best model at or under it. Faded models are dominated; whiskers are the score 95% CI. Prices are hand-maintained snapshots of provider list prices, blending each provider’s input and output rates 3:1, (3 × input + output) / 4 per million tokens. Hover a model for exact numbers.
We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.
