Inside BullshitBench: AI Models and Nonsense Detection

Every question uses real terminology in a structure that looks legitimate. Click through to see the question, a model that caught it (green), and one that didn't (red):

Each response is graded by three judge models (Claude Sonnet 4.6, GPT-5.2, Gemini 3.1 Pro) into one of three outcomes:

• Green – Clear pushback: the model identifies the question as incoherent and declines to engage.
• Amber – Partial recognition: the model flags some issues but still engages with the premise.
• Red – Accepted nonsense: the model treats the question as valid and provides a confident answer.

The chart below shows the response distribution for each tested model, sorted by the proportion of clear pushback (green). Green means the model rejected the nonsense, amber means partial recognition, and red means the model answered as if the question were valid.

Detection rates range from 2% to 91% across the 88 configurations tested. Results by provider:

• Anthropic – dominates the top of the ranking. Claude Sonnet 4.6 leads at 91%; the top 11 positions are all Claude models, down to Claude Sonnet 4.5 at 74%.
• Qwen (Alibaba) – Qwen 3.5 (open-source, 397B MoE) reaches 78%. The highest-ranked non-Anthropic result.
• xAI – Grok 4.20 multi-agent beta reaches 67% (single-agent: 56%).
• OpenAI – GPT-5.4 at 48% is their best result. o3 scores 26%. GPT-4o-mini scores 2%.
• Google – Gemini 3 Pro at 48%. Gemini 2.5 Pro at 20%. Gemma 3 27B at 3%.
• Others – DeepSeek V3.2 (13%), Mistral Large (2%), ERNIE 4.5 (4%). Most smaller or less well-known models fall below 20%.

Many models now support extended "thinking" modes that use more tokens to reason through a problem. The scatter below plots average reasoning tokens per response against clear pushback rate. More thinking does not reliably produce better detection.

For several model families, enabling reasoning actively reduces detection. The chart below shows only models where reasoning made performance worse, comparing their best non-thinking variant to their best thinking variant:

A possible explanation: reasoning training optimizes models to solve whatever problem they are given. More thinking budget means more effort spent constructing an answer, not more scrutiny of whether the question deserved one.

OpenAI and Google models have not improved much on this task across generations. OpenAI went from 12% (GPT-4o) to 48% (GPT-5.4); Google from 15% (Gemini 2.0 Flash) to 48% (Gemini 3 Pro), then back down to 37% (Gemini 3.1 Pro). Earlier Claude models were already ahead of the field, but the 4.5 and 4.6 series moved into a different league entirely: Claude Opus 4.1 scored 43%, then Claude Sonnet 4.5 jumped to 79%, and Claude Sonnet 4.6 reached 91%.

Larger models tend to perform better: there is a positive correlation between parameter count and detection rate among models with publicly known sizes. But size does not explain everything. Qwen 3.5 (397B total, 17B active) reaches 78%, well above what the trendline would predict. DeepSeek V3.2 (685B) scores only 13%, and ERNIE 4.5 (300B) scores 4%.

For anyone using AI models:

• Do not assume a confident, well-structured answer means the question was valid. Most models will produce detailed responses to complete nonsense. If you are relying on AI for research, analysis, or decision-making, the model is unlikely to tell you when your question does not make sense.
• Enabling "reasoning" or "thinking" modes does not fix this. For most model families outside of Anthropic, extended reasoning made nonsense detection worse, not better.

For model developers:

• Nonsense detection appears to be a trainable capability, not an emergent property of scale or reasoning. Anthropic's Claude 4.5 and 4.6 series dramatically outperform all other providers, suggesting that specific training choices or alignment approaches can produce models that reliably challenge broken premises rather than comply with them.

BullshitBench v2 contains 100 nonsense questions across 5 domains (software, finance, legal, medical, physics) using 13 nonsense techniques. An earlier v1 contained 55 general-domain questions. We tested over 80 models from 16 providers, each at multiple reasoning levels (no reasoning, low, high, xhigh where supported).

Model-as-a-judge. Three judges (Claude Sonnet 4.6, GPT-5.2, Gemini 3.1 Pro) independently grade every response as green (clear pushback), amber (partial), or red (accepted nonsense). The final score is the mean of the three. The judges agree on the outcome roughly 80% of the time.

• GitHub Repository – full dataset, questions, code, and benchmark pipeline
• Interactive Viewer – explore per-model responses, domain breakdowns, and technique-level performance
• YouTube Walkthrough – video overview of the benchmark and key findings
• Peter Gostev – author of the benchmark