MethodologyAnnotator qualityUpdatesHugging Face416.9k downloadsSuggest a model or benchmarkRequest a private benchmark
Benchmarks
Dimensions
Models
Categories
composition
tags
View
Image-to-3D updated 2026-10-06

The Human-Preference Leaderboard for Image-to-3D Generation

Rapidatabenchmark@rapidata.ai

Abstract

This benchmark ranks image-to-3D models on how faithfully they rebuild an object as a textured 3D asset from a single image. The rankings come from 61,380 pairwise matches, decided by 306,800 votes. Quality is evaluated in two dimensions: Geometry and Faithfulness. All rankings are fitted with a Bradley–Terry model and reported on the Elo scale. All data and judgments are public; see the details for more.

1  Methodology

We evaluate image-to-3D models purely on human feedback. Humans perform inconsistently on absolute judgements, but much better in a comparative setting. Our rating is therefore based on pair-wise matches, the results of which are jointly fit using the Bradley–Terry model where the probability of model m beating model m′ is given by:

P(m ≻ m′) =eβmeβm + eβm′=11 + e−(βm − βm′)(1)

A match pairs the 3D assets two models generated from the same input image. As evaluating all permutations is infeasible, matches are sampled, ensuring broad coverage per model. The resulting scores are scaled to the familiar Elo range via Rm = 400 βm + 1000 (2), which stretches standard Elo by ln 10 ≈ 2.3 to better highlight differences in model capability.

We curated 357 input images, each showing an object to reconstruct, held fixed to ensure reproducibility of the results. Every asset is shown to annotators as a turntable video rendered identically for every model, next to the input image.

To ensure trustworthy ratings we employ several measures, notably annotator pre-filtering via gold-standard 3D comparisons, and redundancy, collecting a minimum of m responses per match (m = 5) and deciding the winner by majority. Read more

LeaderboardCompare OutputsMatchupsAnalysisInput imagesDetails
1. Source material
The 357 input images come from four sources. Amazon Berkeley Objects (ABO, CC BY 4.0) supplies 103 real catalogue photographs (31%); we restricted it to products without a ground-truth 3D mesh to reduce the risk of training-set contamination. 3DGen-Bench (MIT) supplies 90 generated images of single objects and of groups of objects (27%). FLUX-Reason-6M (Apache 2.0) supplies 26 images generated with FLUX.1 [dev] (8%). The remaining 110 images (33%) were generated for this benchmark with OpenAI GPT Image, with invented brands and text wherever a label or logo appears. Every image carries its own credit line in the released data.
2. Evaluation dimensions
Each input image in the fixed set is evaluated separately across two dimensions. Each vote addresses only one dimension, using the dedicated question shown below. (The word “photo” refers to the input image, which is either a photograph or a generated image.)
Dimension
Question asked to annotators
Geometry
Which video reflects the shape of the object in the photo?
Appearance
Which video looks more like the object in the photo?
Both dimensions show the same pairs of assets beside the same input image and differ only in how the asset is rendered. Geometry shows every asset in one flat grey with its textures removed, so the vote turns on shape alone. Appearance shows the asset as the model delivered it, textures and materials included, so it scores how closely the whole object matches the image. A model that gets the shape right and the colours wrong therefore places higher on Geometry than on Appearance.
3. Input image set version
We curated 357 input images from four sources. ABO supplies catalogue photographs of products that ship no 3D model, and 3DGen-Bench and FLUX-Reason-6M supply generated images. We generated the rest with GPT Image, including original game characters and creatures from Indian, Chinese and Japanese mythology. We chose these sources because no evaluated model is known to have trained on them. A product without a 3D model cannot have been a training target, and a generated image shows no real object that anyone could have scanned.Each image carries two kinds of tag. Composition tags record whether an image shows a single object (261 images) or several (96), and standings are reported for each group separately. The remaining tags mark six properties that make single-image reconstruction hard: thin structures, holes and concavities, occlusion by a second object, transparent or reflective surfaces, fur, hair or foliage, and printed text or logos.Every image is prepared the same way. Transparent areas are flattened onto white, and each image is padded to a square and scaled down to at most 1024 pixels on a side. We keep the set fixed so that results can be reproduced, and every image may be redistributed and is credited individually.
4. Model generation protocol
Each model receives the same single image and runs with a fixed configuration, including whatever preprocessing its provider applies, such as background removal. SAM 3D Objects is the only model that has to be told which object to segment, so it also receives a short noun naming the object. This sometimes results in wrong segmentations that affect the produced asset. Every model produces one asset per image, and we never choose among several outputs. For each model we record the provider, the endpoint, every generation setting and the generation date. Four Hi3D v3.0 assets were generated through Atlas Cloud, with the same model and settings, after the fal endpoint failed on those images.All assets go through one rendering pipeline, so differences between turntables come from the models. Each mesh is centred and scaled uniformly to a fixed size, then rendered as a ten-second turntable (240 frames at 24 fps, 720x720) with the same camera path, field of view, lighting and background for every model. Each turntable starts from the viewpoint of the input image, so its first frame can be compared directly with the image, and the rotation then shows the sides the image could not. Geometry uses a grey turntable rendered from the same file with all textures removed, so no second generation is involved.
5. Matchmaking
Matchups are sampled from the full pool, and every ranked model takes part in at least 6,573 matches.
6. Judgements
This round aggregates 61,380 matches. Each match is decided by a majority of at least five judgements from different annotators.
7. Annotator audience
This round was judged by a dedicated audience rather than Rapidata’s global pool. To join, an annotator answers gold-standard comparisons of 3D turntables, drawn from 44 pairs with a known correct answer on both the grey and the textured renders. Annotators are admitted once they reach 75% accuracy over at least ten of these pairs.
8. Public artifacts and limitations
No dataset has been released for this round yet, so the standings, matchups and input images on this page are everything we publish. All rankings come from human preference judgements by the audience described above, and no automated metric is used.

Full protocol in the global methodology.

We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.

Benchmarks
Text-to-ImageOutput Diversity (Mode Collapse)Multi-Step Image Editing (Identity Drift)Camera MovementRelative Camera MovementPhysicsPhysicsRealismImage-to-3DSVGLLM-to-3DSuggest a model or benchmark
Method
MethodologyData on Hugging Face
Rapidata
rapidata.ai Private benchmarksContactX LinkedIn
Rapidata, benchmark.ai © 2026