AI models agreed with each other more than with reality in a new test
AI models agreed more closely with one another than with laboratory measurements in a new test of scientific predictions, though pooling their rankings improved results on average.
The study, led by UniPat AI and submitted to arXiv on November 11, tested 13 agent configurations spanning 12 models on 100 public tasks and 16 private tasks. Each configuration combined a model, reasoning setting and agent software, asking them to rank experiments across chemistry, materials science, biology and medicine.
Researchers reported that on public tasks, average agreement between scored configurations reached a Spearman rank correlation of 0.52, compared with just 0.33 against measured outcomes. Agreement between agents exceeded their correlation with measurements on 81 of the 100 tasks.
Consensus improved predictions
Despite the discrepancies, pooling predictions showed benefits. Averaging each experiment's position across configurations produced a consensus ranking that scored 0.08 points above the average submission, improving results on 73 of the 100 public tasks. The 95% confidence interval for this gain was 0.05 to 0.11.
Agreement contains useful signal, but shared errors can survive a consensus.
Study limitations
The configurations had unequal coverage across tasks. For example, Claude Opus 5.5 and Claude Fable 5.1 had scores on only 44 and 30 public tasks respectively, largely due to safety refusals. Across the full benchmark, all 13 configurations shared just 28 scored tasks.
The private set contained only 16 tasks across four unpublished experimental series, limiting how broadly its findings can be applied. Notably, twelve of the 13 configurations ranked the unpublished experiments significantly better than chance overall.
The study used no human prediction baseline, and its findings apply specifically to ranking the experimental outcomes it tested, with laboratory measurements still needed to verify the answers.
Related
