AutoEval provides a Day-1 signal as soon as a model launches. Scores are clearly labeled “AutoEval,” then updated once enough human votes arrive to validate them. This helps the community identify the best models for their tasks, sooner.
How do AutoEval scores work?
Instead of waiting on votes, we train a Reward Model (RM) that captures human preference, then use it to cast votes automatically. By integrating live human votes with these RM-generated proxy votes, we unlock:
- Rapid feedback: RM-based voting gives a ranking in under an hour, not days.
- High flexibility: Because we control the prompts the RM votes on, we can target evaluation on specific domains.
Arena AutoEval works across text, vision, image generation, and coding domains, supplementing our human evaluation with rapid, targeted automation where just human voting cannot scale quickly enough.
Reward Model Training
AutoEval starts with data. We train a reward model on Arena’s large-scale human preference dataset, consisting of millions of pairwise comparisons from live votes, so that it learns to approximate human judgments. At its core, the model is a pointwise reward model: it maps each individual prompt-response pair to a scalar score. We train it on pairwise comparison data so that the resulting score rankings consistently align with human preferences.
Learning from massive real preference data pays off. Our text reward model predicts human preferences 8–10% more accurately than frontier LLM judges (Gemini-3-flash/pro, GPT-5). Many prompts on Arena are hard and complex, and the competing models are strong, so there’s often no single response that clearly dominates in all aspects. In those cases, we found that LLM judges struggle to make the final call, while our reward model still captures what humans actually prefer.
Training a reward model for a live service is not a one-time learning problem. The target, the data, and the models being evaluated all continue to change. We encountered several challenges:
- Human preferences are a moving target. What users consider a good response changes as expectations, interaction styles, and use cases evolve. Historical feedback provides breadth and stability, while recent feedback reflects these shifts. Effective training must grounded in the vast historical feedback signal while adapting to the recent trend drifting over time.
- The model frontier keeps moving. As models improve, obvious failures become less common and quality differences become more subtle. Reward models must keep pace with emerging capabilities and remain sensitive to increasingly fine-grained distinctions.
- Preference labels contain genuine ambiguity. Human votes are not always decisive: different users value different qualities, and ties may represent either two similarly strong responses or two similarly weak ones. Treating every comparison as a simple win or loss loses this information, so ties and calibration need to be modeled explicitly.
- Multi-turn feedback is context-dependent. A final vote may reflect earlier decisions, context management, or intermediate results, making it difficult to isolate which behavior drove the preference.
- Complex tasks require hybrid signals. For difficult domains like coding, an answer can’t be judged on a single metric. Every aspect—including code correctness, technical quality, the look of the final interface, and the overall user experience—has to be taken into account.
With these challenges carefully handled, our reward model is accurate and reliable enough to be useful in practice.
From Reward Model to AutoEval
AutoEval uses the reward model in place of human votes, then follows the exact same Arena ranking methodology to compute the final arena score. Specifically, for every target model it runs four steps:
- Live Eval Sampling — sample real prompt-and-response pairs from live evaluation.
- Model Generation — regenerate response B from the target model on the same prompt.
- RM Pairwise Vote — the reward model scores each response, applying a softmax to the score differences to obtain a soft probability vote.
- AutoEval Score — combine the autorater’s votes for the target model with the live human votes for other models to produce the Arena Score.
So, how close is AutoEval to the real thing? To test this, we trained a reward model using data up to the end of April 2026 and evaluated AutoEval scores for all new models released after that date. When comparing these to live evaluation scores, the results (shown in the figure below) demonstrate that AutoEval scores are highly correlated with live evals. Specifically, the ranking correlation exceeds 0.98, and most AutoEval scores fall well within the confidence intervals of the live scores.
We also tested AutoEval’s ability to correctly choose the better model in head-to-head comparisons across >40 test models. AutoEval achieves >90% accuracy when the true performance gap exceeds 10 points, and 100% accuracy beyond 15 points. When models are within 5 points of each other, they are statistically indistinguishable under live evaluation due to overlapping confidence intervals.
This high correlation demonstrates that AutoEval is accurate enough to drive critical model selection decisions. For example, it can be used to shortlist the most promising models from a large pool before committing resources to a full live evaluation.
Beyond text: one method, many modalities
AutoEval isn’t text-only. We apply the same methodology across modalities — text, vision, image generation, and code arenas.
Taking image generation as an example, we trained a text-to-image reward model on over 3 million preference pairs. By training on this massive preference dataset—built from realistic image generation prompts requested by real-world users—our reward model not only excels on our own distribution but also achieves top-tier performance on public benchmarks. Evaluated on MMRB2, Meta’s 2026 multimodal reward benchmark, Arena-RM achieves state-of-the-art compared with other pointwise text-to-image reward models, more than 9 points ahead of the runner-up (HPSV3, UnifiedReward, PickScore, VQAScore and others).
What’s next
Arena leaderboards remain a public good and AutoEval makes that value available sooner: right at a model's release. It already works across Text, Vision, Image, and Code Arena, providing the ability to rank new models in hours rather than days, while focusing on the domains the community cares about. As we continue working through open challenges in reward modeling, we’ll be extending this approach to even more modalities and incorporating more signals to better capture human preferences.
