All three iterations of Claude 3 secure positions in the top ten.
Claude 3 Opus, the latest artificial intelligence model developed by Anthropic, has claimed the top spot on the Chatbot Arena leaderboard, displacing OpenAI’s GPT-4 to second place for the first time since its inception last year.
Unlike conventional methods of assessing AI models, the LMSYS Chatbot Arena relies on human judgments, where individuals rank the outputs of two different models generated from the same prompt.
OpenAI’s various versions of GPT-4 have maintained dominance for a considerable duration, leading any model that approaches its benchmark scores to be deemed a GPT-4-class model. Perhaps a new classification, the Claude-3 class, needs introduction for forthcoming evaluations.
It’s noteworthy that the gap in scores between Claude 3 Opus and GPT-4 is minimal, considering the latter has been in existence for a year, and an anticipated GPT-5, described as “markedly different,” is expected sometime this year—potentially challenging Anthropic’s current position.
Understanding the Chatbot Arena:
The Chatbot Arena, managed by LMSys, the Large Model Systems Organization, orchestrates diverse large language models competing anonymously in randomized battles.
Launched initially in May last year, the platform has amassed over 400,000 user evaluations, predominantly featuring models from Anthropic, OpenAI, and Google among the top contenders throughout its operation.
Recent entries from other models, such as those from French AI startup Mistral and Chinese enterprises like Alibaba, have begun to ascend the rankings, while open-source models are increasingly prevalent.
Here is the revised ranking table:
| Rank | Model | Elo | Votes |
|---|---|---|---|
| 1 | Claude-3 Opus | 1253 | 33250 |
| 1 | GPT-4-1106-Preview | 1251 | 54141 |
| 1 | GPT-4-0125-preview | 1248 | 34825 |
| 4 | Gemini Pro | 1203 | 12476 |
| 4 | Claude-3 Sonnet | 1198 | 32761 |
| 6 | GPT-4-0314 | 1185 | 33499 |
| 7 | Claude-3 Haiku | 1179 | 18776 |
| 8 | GPT-4-0613 | 1158 | 51860 |
| 8 | Mistral-Large-2402 | 1157 | 26734 |
| 9 | Qwen1.5-72B-Chat | 1148 | 20211 |
| 10 | Claude-1 | 1146 | 21908 |
| 10 | Mistral Medium | 1145 | 26196 |
The Chatbot Arena employs the Elo rating system, commonly utilized in games like chess, to assess the relative proficiency of players. However, in this context, the ranking applies to the chatbot itself rather than the human user interacting with the model.
The Chatbot Arena, while insightful, has limitations. It doesn’t include every LLM, potentially missing hidden gems. Additionally, some models might have outdated versions included, and technical issues like GPT-4 loading problems can skew user evaluations. Live internet access for models like Gemini Pro might also create an unfair advantage for tasks requiring real-time information. Finally, the arena focuses on conversation, neglecting other crucial LLM skills like factual accuracy or code generation. Considering these limitations helps us interpret the rankings with a more nuanced perspective.
Notably absent from the arena are some prominent models, like Google’s Gemini Pro 1.5, renowned for its extensive context window, and Gemini Ultra.
Highlighting Performance and Progress:
The latest update, fueled by over 70,000 new votes, saw Claude 3 Opus ascend to the leaderboard’s pinnacle. Even the smallest variants of the Claude 3 series showcased commendable performance.
LMSYS provided insight, remarking on Claude-3 Haiku’s remarkable performance, likening it to GPT-4 in terms of user preference. Despite its “local size” model status, akin to Google’s Gemini Nano, Haiku exhibits unparalleled speed, capabilities, and context length.
What’s particularly noteworthy is Haiku’s achievement despite its relatively modest scale compared to Opus or GPT-4-class models. While not as intellectually robust as Opus or Sonnet, Anthropic’s Haiku offers notable advantages in terms of cost-effectiveness and speed, matching larger models in blind-tests, as indicated by arena results.
Observations on Model Distribution:
All three variants of Claude 3 secure positions in the top ten, with Opus leading the pack, Sonnet tied for fourth with Gemini Pro, and Haiku sharing sixth place with an earlier iteration of GPT-4.
The dominance of proprietary models in the top 20 of the arena leaderboard suggests that open-source initiatives have ground to cover to compete with industry giants.
Anticipated developments include Meta’s forthcoming release of Llama 3, expected to join the top tier of models. Meta’s vast computational resources, comprising over 300,000 Nvidia H100 GPUs, indicate its potential to rival Claude 3 in capability.
In parallel, the industry sees shifts toward open-source and decentralized AI, with StabilityAI’s founder, Emad Mostaque, stepping back from CEO responsibilities to champion more distributed and accessible artificial intelligence. Mostaque advocates for decentralized approaches, highlighting the limitations of centralized AI models.
This is fascinating stuff. Claude 3 Opus dethroning GPT-4 is a big deal. It’ll be interesting to see how OpenAI responds with their rumored “markedly different” GPT-5. The race for chatbot supremacy is heating up.
The chatbot Arena is interesting, but it only focuses on conversation. What about factual accuracy or code generation? These are crucial LLM skills too. We shouldn’t judge a book by its cover, or a chatbot by its conversational skills alone.
Cyber_sleuth brings up a valid point. The Arena rankings are a good starting point, but they don’t tell the whole story. However, claude 3’s across-the-board strong showing, with all three variants ranking highly, suggests they’re doing something right
Claude 3 Haiku’s performance is particularly impressive. A “local size” model matching GPT-4 in user preference? That’s a game-changer. Imagine the cost-effectiveness and speed benefits if this translates to real-world applications
Claude 3 Opus on top? I knew those Anthropic guys were cookin’ up something special. Can’t wait to see how GPT-5 shakes things up though. This chatbot arms race is getting intense
I want an AI that can beat me at chess and write a sonnet about it.
This whole ranking thing is silly. AI is about more than just chatting or writing poetry. We need them tackling real-world problems, like climate change or healthcare. Let’s see the Chatbot Arena throw those kinds of challenges at them.
Open source where you at? These big companies are hoarding all the good stuff. We need more accessible AI, not another corporate overlord.