LMArena, originally launched as Chatbot Arena, is a crowdsourced evaluation platform that ranks large language models and other generative models through blind, pairwise comparisons voted on by the public. Users submit a prompt, receive responses from two anonymized models, and vote for the better answer; those votes feed an Elo-style rating system that produces a continuously updated leaderboard. The project originated at the University of California, Berkeley, from the same research group behind the Vicuna open-weights model, and later spun out as an independent company.
History
Chatbot Arena launched in 2023 as a research tool for comparing instruction-tuned chat models, at a time when standard benchmarks like MMLU were increasingly seen as saturated or vulnerable to training data contamination. Because the arena relies on live human preference rather than static test sets, it was adopted quickly by both open-source developers and major labs, who began citing arena rankings alongside formal benchmark scores when announcing new models. The project rebranded to LMArena in 2025 as it expanded beyond text chat into image, video, and coding-specific arenas, and as its Berkeley-affiliated founders formed a company to operate the platform independently while keeping the leaderboard freely accessible.
Methodology and influence
LMArena's ranking system uses an Elo formula adapted from chess and other competitive-rating contexts, updated as new votes arrive, along with statistical controls for factors such as response style and length that critics noted could bias human raters toward more verbose or stylistically confident answers regardless of correctness. Style-control variants of the leaderboard were introduced to address this. Labs including OpenAI, Google DeepMind, Anthropic, xAI, and DeepSeek have all featured prominently on the leaderboard, and a strong arena ranking became a marketing point for new model launches such as Gemini, Grok, and DeepSeek-R1.
Criticism
LMArena has faced criticism that labs can submit multiple unreleased variants of a model to the arena to select the best-performing checkpoint before public release, effectively optimizing for the test rather than for general quality, an accusation that surfaced prominently around Meta's Llama 4 launch in 2025. Researchers have also questioned whether crowdsourced aesthetic preference, dominated by a self-selected pool of enthusiast voters, correlates well with real-world task performance measured by benchmarks like SWE-bench or HumanEval. Despite these critiques, LMArena remains one of the most widely cited public reference points for comparing model quality across the industry.