Annotator quality

Annotators reach tasks inside third-party apps and advertising contexts, where a short validation task appears in place of a conventional ad. This is how benchmark.ai collects millions of judgements per day from a broad, non-expert population rather than a small panel.

A new annotator is first shown validation tasks only. Once they answer enough correctly they graduate into the global audience and their judgements begin to count. The funnel below shows the stages; no annotator contributes scored judgements before graduating.

A validation task is a task with a known correct answer, indistinguishable to the annotator from a scored task. Validation tasks are interleaved continuously, not only at onboarding, so reliability is measured throughout an annotator's activity.

The global audience is the pool of annotators who have graduated and remain in good standing. Benchmarks draw from this audience by default, giving a broad, demographically diverse population rather than curator taste.

Each annotator carries a user score updated continuously from their validation-task performance and other signals. The user score reflects how much a given annotator's judgements can be trusted.

Annotators below a reliability threshold are excluded, and their judgements are removed retroactively from affected matches. Thresholds trade coverage against noise; the estimator itself is proprietary and treated here as a black box.

Beyond the global audience, a curated audience restricts to demographics or expertise, and a custom audience is defined per project. Public benchmarks use the global audience unless a round's protocol states otherwise.

Every pairwise match is shown to several distinct annotators (by default five). Independent judgements reduce the effect of any single noisy response; the match winner is the majority.

We compute both a raw aggregation (equal weight per judgement) and a reliability-weighted aggregation (weighted by user score). Published leaderboards use the reliability-weighted result; the raw result is available for audit.

Uncertainty is estimated by bootstrapping the judgement set (1,000 resamples) and reporting 95% intervals, so every score carries an explicit interval rather than a bare point estimate.

Judgements, prompts, and outputs are released publicly so third parties can recompute rankings and inspect the raw votes behind any match. Reproducibility is the core trust mechanism.

Self-reported demographics vary per round because the audience is sampled fresh.