The MultipleChat
AI Benchmark

MultipleChat runs a collaborative mode in which a user's prompt is sent to multiple frontier models in parallel, and the responses are displayed side-by-side. The user then chooses which response to continue with, copy, or refine. This produces a continuous stream of behavioral preference data — users vote with their workflow, not with a survey.

Q2 2026 covers all collaborative-mode comparisons between April 1 and June 30, 2026, where at least three frontier models returned valid responses to the same prompt. Total: 50,142 comparisons across 11 task categories from ~8,400 unique users. {{MOCK}}

Task categorization was performed automatically by Gemini Flash, then spot-validated on a 500-prompt sample (κ = 0.81 agreement with two human reviewers). Categories: writing, coding, summarization, translation, reasoning, factual Q&A, math, research, creative, conversational, structured extraction.

Win criterion. A "win" is recorded when the user (a) continues the conversation thread with that model's response, OR (b) explicitly copies/saves that response within 90 seconds of receiving the parallel outputs. Ties are split. Prompts where no continuation occurred within 5 minutes are excluded from win-rate calculations but retained for hallucination and latency metrics.

Hallucination flagging. A response is flagged as a hallucination candidate when (a) it contains a citation or factual claim that fails programmatic verification against authoritative sources, OR (b) the user invokes the "regenerate" function with a prompt indicating factual concern (e.g. "that's wrong", "fact check"). Flags are reviewed for false-positive rate on a 200-response random sample.

What we exclude. Prompts containing PII, prompts under 8 tokens, prompts where any model errored mid-stream, and the first 100 comparisons of any new user (warmup bias). The raw prompt text never leaves MultipleChat's servers — only the aggregated outcome.

Grok dominates this chart. At ~$5.90 per 1,000 wins, it costs roughly half the average and 40% less than the most expensive model. For a team running 100k queries/month, the swing between always-Claude and Grok-default-with-fallback is approximately $11,000/month in inference cost.

The expensive answers are the rare ones. Perplexity Sonar's $/win looks high because it only competes on research tasks (where its win rate is 51%); on a per-task-it-actually-wins basis, it is competitive.

The reasoning-model paradox. Independent research showed OpenAI's o3 reasoning model hallucinating on the PersonQA factuality benchmark at 33% — more than double its non-reasoning predecessor o1 at 16%. Our data replicates the pattern inside MultipleChat: when users send simple factual questions to reasoning-tier models, hallucination flags rise 2.1× compared to non-reasoning siblings. {{MOCK}}

Where this matters. Don't use reasoning models for atomic factual lookup. Use them for multi-step problems where their thinking is the value. The same model that solves a hard math proof correctly will confidently misremember a CEO's name.

Domain risk persists. Independent 2026 studies report 6.4% hallucination in legal queries, 10–20% in medical, and up to 33% in RAG-based legal tools. Our equivalent flags in the legal/medical categories of our dataset sit at 7.2% and 12.4% respectively. Use these models for drafting, not for final-step diagnosis. {{MOCK}}

Self-selection bias. Our users are productivity-oriented professionals using a multi-model platform — they may differ from the general public. We over-index on coding, writing and research; under-index on casual chat and roleplay.

The "win" signal is behavioral. A user continuing with model X's response means that response was good enough to use, not necessarily the best possible. We trade verifiability against authenticity — the LMSYS arena gets more deliberate votes, we get more honest ones.

Model versions move fast. "Claude" in this dataset means whatever Anthropic was serving on each measurement day — we record specific build IDs in the downloadable dataset. The headline numbers will shift when new versions ship.

We sell access to these models. MultipleChat charges for access to all five vendors and has no contracted preference. We have an obvious incentive to make the case for a multi-model platform — which we explicitly do. We do not have an incentive to favor any single vendor.