Research note · Evaluation
The locked accuracy test: what a council of models does and doesn't add
· Danphe Intelligence Lab · 7 min read
A council of AI models costs more than one model, so it has to earn its place. We froze the hypotheses, rules, configuration and data before running a single locked question, and publish everything, including the hypothesis that failed.
- Smart mode accuracy
- 85.1%
- 483 unseen questions
- Claude Opus 5.5 alone
- 83.4%
- statistical tie, p = 0.13
- Model cost per question
- $0.236
- limit was $0.279
- Hypotheses passed
- 3 of 4
- long answers failed
The set-up
We wanted to know two things: is the council at least as accurate as the best single model, and what does it add beyond accuracy? Everything was frozen on 5 November 2026. Nothing was re-run or tuned afterwards.
- Questions: 483 unseen questions across five domains (contracts (ContractNLI), science papers (QASPER), biomedical (PubMedQA), broad reasoning (MMLU-Pro) and graduate science (GPQA Diamond)), sampled once with a fixed seed and never used for tuning. Plus 100 long-form research-paper questions.
- Systems: Danphe Smart mode as shipped (two models answer; the full council runs when they disagree), Claude Opus 5.5 alone, and GPT-6.1 Sol alone.
- Same conditions: the same documents, prompt bundle and strict parser for every system. Failures and timeouts count as wrong.
- Statistics: exact McNemar paired tests (two-sided, α = 0.05) and 95% Clopper–Pearson intervals.
The four hypotheses
Written down before the run, with their decision rules.
H1 · No worse than the best single model (passed)
Smart mode 85.1% vs Claude Opus 5.5 alone 83.4%: 15 questions only Smart got right, 7 only Opus got right (exact McNemar p = 0.13). Inside the 2-point margin.
H2 · Better than GPT-6.1 Sol alone (passed)
Smart mode 85.1% vs 81%: 30 vs 10 discordant questions, p = 0.002.
H3 · Half the cost of the full council or less (passed)
$0.236 per question against a preregistered limit of $0.279 (50% of the Standard council's development mean).
H4 · As precise as the best single model on long answers (failed)
Smart mode's statement precision was 2.1 points below GPT-6.1 Sol alone (limit: 2), and its share of answers with a contradiction (5.3%) was above Opus's (4.4%).
Accuracy on 483 unseen questions
Five domains, locked test, November 2026. Dot = accuracy; bar = 95% confidence interval. Paired tests are against Danphe Smart mode.
Danphe Smart mode
Reference
0.0%
Claude Opus 5.5 alone
p = 0.13 · statistical tie
0.0%
GPT-6.1 Sol alone
p = 0.002 · Danphe significantly better
0.0%
Preregistered on 5 November 2026; exact McNemar paired tests; failures count as wrong. Average model cost $0.24 per question. How we measured
Per domain
No single-domain difference is significant at about 100 questions each, so we make no per-domain claim.
| Domain | Opus alone | GPT-6.1 Sol alone | Smart mode | Cost / question |
|---|---|---|---|---|
| Contracts (ContractNLI, 100) | 84.0% | 85.0% | 86.0% | $0.48 |
| Science papers (QASPER, 100) | 87.0% | 80.0% | 86.0% | $0.27 |
| Biomedical (PubMedQA, 100) | 68.0% | 63.0% | 72.0% | $0.19 |
| Broad reasoning (MMLU-Pro, 100) | 89.0% | 86.0% | 89.0% | $0.10 |
| Graduate science (GPQA Diamond, 83) | 90.4% | 92.8% | 94.0% | $0.12 |
Contracts ran in a high-risk legal workspace, which always uses the full council; that is why it costs more.
Long answers: the hypothesis that failed
On 100 research-paper questions that need a detailed answer, three AI judges from three model families checked every statement against the paper. GPT-6.1 Sol alone was the most precise writer: 95.1% of its statements were supported, with no answer containing a contradiction. Smart mode reached 93%, about level with Claude Opus 5.5 alone (92.6%). H4 failed.
- The judges disagree on the ranking. DeepSeek V3.2 ranked Smart mode first; Claude Sonnet 5.5 and GPT-6.1 Terra ranked GPT-6.1 Sol first. Our rule only calls a result robust when all three agree.
- One judge's coverage is incomplete. GPT-6.1 Terra returned verdicts for only 52–54 of 100 answers per system.
- Smart mode escalated 90 of 100 long questions to the full council, because two free-text answers rarely match word for word.
What the council adds
On top of top-model accuracy, every statement is checked against your documents with an exact quote, disagreement between models is shown instead of hidden, and every answer has a signed audit record. Where a dataset marks the passage that holds the answer, 100% of the council's quotes landed on it for contracts and 82.1% for science papers.
What we learned
- 1On short, checkable questions the strongest frontier model is already very good. Smart mode keeps that accuracy, wins slightly more often than it loses, and is significantly better than a weaker frontier model.
- 2On long answers the deciding factor is the writer. GPT-6.1 Sol writes more cautiously; our final answer is written by Opus as editor, so it inherits Opus's style.
- 3Asking one model five times and voting never helped in our earlier tests: errors are systematic, not random.
What we will and won't say
We say
Smart mode matches the best single model on unseen questions across five domains and is significantly more accurate than a leading alternative, at $0.24 a question on average.
We don't say
That it is more accurate than Claude or GPT, or that it has fewer hallucinations on long answers. “Confirmed” labels wait for a signed-off calibration.
Next, we will test the most precise long-form writer as the editor on fresh data, and run larger locked tests (300+ per domain) before any public “beats” claim. The full numbers are on the Accuracy page.