Accuracy
Measured, not promised
We publish what we measured, how, and where Danphe does not come out ahead. On short questions Danphe matches the best single model and beats a weaker one; on long answers a single model was more precise. The clearer gain is that every statement is checked and quoted.
Locked test
Headline result
483 questions the system had never seen, across contracts (ContractNLI), science papers (QASPER), biomedical (PubMedQA), broad reasoning (MMLU-Pro) and graduate science (GPQA Diamond). Run in November 2026 under rules preregistered on 5 November 2026.
Accuracy on 483 unseen questions
Five domains, locked test, November 2026. Dot = accuracy; bar = 95% confidence interval. Paired tests are against Danphe Smart mode.
Danphe Smart mode
Reference
0.0%
Claude Opus 5.5 alone
p = 0.13 · statistical tie
0.0%
GPT-6.1 Sol alone
p = 0.002 · Danphe significantly better
0.0%
Preregistered on 5 November 2026; exact McNemar paired tests; failures count as wrong. Average model cost $0.24 per question. How we measured
Show as a table
| System | Accuracy | 95% CI | Paired test |
|---|---|---|---|
| Danphe Smart mode | 85.1% | 81.6–88.1% | Reference |
| Claude Opus 5.5 alone | 83.4% | 79.8–86.6% | p = 0.13 · statistical tie |
| GPT-6.1 Sol alone | 81.0% | 77.2–84.4% | p = 0.002 · Danphe significantly better |
Preregistered tests
Written down and committed before any locked run. Three passed; one failed.
- Pass
H1 · No worse than the best single model (Claude Opus 5.5)
15 vs 7 discordant questions, p = 0.13
- Pass
H2 · Better than a leading alternative (GPT-6.1 Sol)
30 vs 10 discordant questions, p = 0.002
- Pass
H3 · Cost at most 50% of the Standard council
$0.236 per question against a $0.279 limit
- Fail
H4 · Long answers: precision within 2 points of the best single model
2.1 points below GPT-6.1 Sol alone
By domain
Results per domain
About 100 questions per domain gives wide intervals, so no single-domain difference is significant. We make no per-domain claims.
| Domain (questions) | Opus 5.5 alone | GPT-6.1 Sol alone | Danphe Smart | Smart cost / question |
|---|---|---|---|---|
| Contracts, ContractNLI (100) | 84.0% | 85.0% | 86.0% | $0.48 |
| Science papers, QASPER (100) | 87.0% | 80.0% | 86.0% | $0.27 |
| Biomedical, PubMedQA (100) | 68.0% | 63.0% | 72.0% | $0.19 |
| Broad reasoning, MMLU-Pro (100) | 89.0% | 86.0% | 89.0% | $0.10 |
| Graduate science, GPQA Diamond (83) | 90.4% | 92.8% | 94.0% | $0.12 |
| Pooled (483) | 83.4% (79.8–86.6) | 81.0% (77.2–84.4) | 85.1% (81.6–88.1) | $0.236 |
Contracts always run the full council (high-risk legal workspace), which is why that domain costs more. The pooled row shows 95% Clopper–Pearson intervals.
Verifiability
Can you check the answer?
Where a dataset marks the passage that contains the answer, we measure whether Danphe's quotes point at it. Single models give no checked quotes, so there is nothing to compare.
100%
of quotes on the marked passage, contracts
ContractNLI, locked test, n = 100 questions.
82.1%
of quotes on the marked passage, science papers
QASPER, locked test, n = 100 questions.
Limits
Where Danphe is not better
These are the results that don't flatter us. They matter as much as the headline.
On long answers, GPT-6.1 Sol alone was the most precise
| System | Statements per answer | Supported by the paper | Answers with a contradiction | Covers the expert answer |
|---|---|---|---|---|
| GPT-6.1 Sol alone | 11.3 | 95.1% | 0.0% | 83% |
| Danphe Smart mode | 13.8 | 93.0% | 5.3% | 83% |
| Claude Opus 5.5 alone | 13.7 | 92.6% | 4.4% | 85% |
- The judges disagree on the ranking: DeepSeek V3.2 ranks Smart mode first; Claude Sonnet 5.5 and GPT-6.1 Terra rank GPT-6.1 Sol first.
- One judge's coverage is incomplete: GPT-6.1 Terra returned verdicts for only 52–54 of 100 answers per system.
- The council's final answer is written by Opus as editor, so it inherits Opus's style. Changing the writer will be re-tested on fresh data before any claim.
It does not beat the best single model on short questions
Smart mode 85.1% vs Claude Opus 5.5 alone 83.4%: the gap is not significant (p = 0.13). When every model makes the same mistake, cross-checking cannot fix it.
It costs more
Smart mode averaged $0.24 per question in model costs, more than asking one model once. A full Standard council costs more again.
It is slower
Several models answer and check each other, so an answer takes longer than asking one model: seconds on the fast path, minutes when the full council runs.
“Confirmed” needs calibration
A statement is only marked “Confirmed” when a workspace has a signed-off calibration. Until then you see “Models agree”, which is backed by checks and a quote but has no measured error rate.
Summary
Danphe or one model on its own?
| What you get | Danphe Smart mode | One top model alone | Who comes out ahead |
|---|---|---|---|
| Accuracy, short questions (483 unseen) | 85.1% | 83.4% Claude Opus 5.5 81.0% GPT-6.1 Sol | Ties the best (p = 0.13); beats GPT-6.1 Sol (p = 0.002) |
| Precision of long answers (100 unseen) | 93.0% of statements supported | 95.1% (GPT-6.1 Sol) | A single careful writer did better |
| Every statement checked against your sources | Yes, with the exact quote | No | Danphe |
| Disagreement between models | Shown, with both readings | Not visible: one voice | Danphe |
| Audit record | Every answer, hash-chained | No | Danphe |
| Model cost per question | $0.24 on average | Lower | Single model |
| Time to answer | Seconds to minutes | Faster | Single model |
Locked tests, November 2026. The long-form test was preregistered and Danphe did not pass it. Read where Danphe is not better
Method
How we measured
- Preregistered. Hypotheses, decision rules, configuration and data were frozen on 5 November 2026, before any locked-test run. Nothing was re-run or tuned afterwards.
- Unseen questions. The locked pools were sampled once (fixed seed) and never used for tuning: ContractNLI 100, QASPER 100, PubMedQA 100, MMLU-Pro 100, GPQA Diamond 83.
- Same conditions. Every system got the same questions, documents and prompt bundle, and was scored by the same strict parser.
- Failures count as wrong. Timeouts and failed runs score zero.
- Statistics. Exact McNemar paired tests (two-sided, α = 0.05) and 95% Clopper–Pearson intervals.
- What was tested. Danphe Smart mode as shipped (two models answer; the full council runs when they disagree), against Claude Opus 5.5 alone and GPT-6.1 Sol alone.
- Long answers were judged statement by statement against the paper by three AI judges from three model families; the table shows their mean.
- Datasets are used under their published licences.
The full write-up, with the hypotheses as preregistered, is in our research note The locked accuracy test. Questions about the method? Contact us.
| Written by ↓ · checked by → | Claude | GPT | Grok | Kimi |
|---|---|---|---|---|
| ClaudeAnthropic | does not check itself | checks | checks | checks |
| GPTOpenAI | checks | does not check itself | checks | checks |
| GrokxAI | checks | checks | does not check itself | checks |
| KimiMoonshot AI | checks | checks | checks | does not check itself |
Plain-language summary