Benchmark
reviewbench: how often we anticipate real referees.
Updated 23 Oct 2026 · 20 held-out papers · eLife Reviewed Preprints
Abstract
We take published life-science manuscripts whose referee reports are public, run our review on the version the referees saw, and count how many of their major concerns our report also raised. On 20 held-out papers the current release anticipates 85% of major concerns. It is weakest on novelty and framing.
- Major-concern recall
- 85%95% CI 80%–89% · gate 85%
- All-concern recall
- 71%95% CI 67%–75%
- Quote fidelity
- 100%Quotes found verbatim
- Run-to-run spread
- ±2 ptsSame config, two runs
Fig. 1Major-concern recall by release. Filled points clear the launch gate; a release does not ship to paid reports until it does.
Fig. 2Recall by concern type for the current release; n is the number of referee concerns of that type in the test set.
Dataset
50 manuscripts from eLife’s Reviewed Preprints, refereed between June and October 2026, across 19 life-science subjects. We split them once: 30 for developing prompts, 20 held out for scoring. Papers we have read the referee reports for stay in the development set.
Each referee report is broken into individual concerns and labelled major or minor by the referee’s own framing, then tagged with one of six concern types.
Scoring
For each referee concern we ask whether any finding in our report raises the same problem — not the same wording. Recall is the share of referee concerns matched.
We also check that every quote in the report appears verbatim in the manuscript. Findings whose quotes cannot be found are removed before the report reaches you, so quote fidelity is 100% by construction; we report how many were removed.
What counts as an improvement
The 95% intervals are bootstrapped over papers, so they reflect how much the score depends on which twenty papers were drawn. Model output also varies between runs, so a new release must beat the current one on the held-out set by more than the run-to-run spread — about 2 points — before we call it better. Releases below the 85% launch gate are not used for paid reports.
Limitations
Referees miss things, and not every concern they raise is right. A finding that matches no referee may still be valid; we track those separately with expert spot checks rather than counting them here.
The set is eLife and life sciences only. Results may not transfer to clinical, physical or computational papers, which we will not offer until they have their own benchmark.