BTCLoading…

AI models agreed with each other more than with reality in a new test

Science Prediction Universe found that pooling rankings improved average results, while shared scientific errors survived. Read the full story on CryptoSlateUnderstand the crypto stories that matter and what to watch…

LW
Liam 'Akiba' Wright
Wire content from CryptoSlate

Quick Take

  1. 01 SPU tested 13 agent configurations, finding stronger agreement between agents than with measured outcomes across public ranking tasks.
  2. 02 Pooling rankings improved scores on 73 of 100 public tasks; the authors say shared errors can remain.
  3. 03 Unequal coverage, a small private sample and no human baseline limit what the study establishes about scientific ability.

Make CryptoSlate logo CryptoSlate preferred on Google logo Share Share

AI models agreed more closely with one another than with laboratory measurements in a new test of AI predictions. Pooling their rankings improved results on average, but several systems could still share the same mistake.

The Science Prediction Universe study, led by UniPat AI and submitted to arXiv on Nov. 11, tested 13 agent configurations spanning 12 models on 100 public tasks and 16 private tasks. Each configuration combined a model, reasoning setting and agent software. The study asked them to rank specified experiments across chemistry, materials science, biology and medicine.

Science Prediction Universe researchers reported that, on the public tasks, average agreement between scored configurations reached a Spearman rank correlation of 0.52, compared with 0.33 against measured outcomes. Agreement between agents exceeded their correlation with measurements on 81 of the 100 tasks. These averages use available scored predictions, with coverage varying by configuration.

Those figures describe how closely orderings match, rather than percentages of answers correct. Agents might rank reaction conditions by yield or materials by conductivity without having to predict numerical values. A perfect ordering without measurement ties scores 1; random rankings average 0.

Related Reading

Google’s new AI reveals hidden genetic clues in new immune-cell test

Consensus improved predictions

The researchers found a benefit to pooling predictions. Averaging each experiment’s position across configurations produced a consensus ranking that scored 0.08 points above the average submission, improving results on 73 of the 100 public tasks. The reported 95% confidence interval for the gain was 0.05 to 0.11.

On one oncology task, eleven configurations agreed at 0.9 but averaged just 0.27 against measurements. The benchmark’s best individual system also beat the consensus of its peers on matched tasks.

More broadly, tasks with greater agreement tended to have better predictions. The paper’s agreement analysis therefore presents consensus as an informative but fallible signal. The authors explain that averaging can reduce errors that differ between systems while retaining common ones.

Agreement helps. Shared errors survive.

Pooling rankings improved average predictions in SPU. Comparing the public-task average with one oncology task shows why agreement still needs an experimental check.

Agreement and measured experimental rankingsPOOLING RANKINGS · SCHEMATICDifferent predictionsAverage positionsavg+0.081score points vs. average submission73 of 100 public tasks improvedDiffering errors can cancel. Shared errors can remain.Agreement and measured experimental rankingsPOOLING RANKINGS · SCHEMATICDifferent predictionsavgAveragepositions+0.081score points vs. average submission73 of 100 public tasks improvedDiffering errors can cancel.Shared errors can remain.

Compare agreement with measured outcomes

Compare agreement with measured outcomesPublic-task averageOne oncology task

Mean pairwise agreement was 0.52, while the mean score against measurements was 0.33, using available scored predictions.

Agreement and measured experimental rankingsAcross 100 public tasksSpearman rank correlationAgreement between agents: 0.52Against measurements: 0.33-1010.520.33Opposite orderSame orderCorrelation is not a percentage correct.Agreement and measured experimental rankingsAcross 100 public tasksSpearman rank correlationAgreement between agents: 0.52Against measurements: 0.33-1010.520.33Opposite orderSame orderCorrelation is not a percentage correct.

Eleven configurations agreed strongly on one oncology task, but their rankings matched the measurements much less closely.

Agreement and measured experimental rankingsOne oncology task · eleven configurationsSpearman rank correlationAgreement between agents: 0.90Against measurements: 0.27-1010.900.27Opposite orderSame orderCorrelation is not a percentage correct.Agreement and measured experimental rankingsOne oncology task · 11 configurationsSpearman rank correlationAgreement between agents: 0.90Against measurements: 0.27-1010.900.27Opposite orderSame orderCorrelation is not a percentage correct.

Agreement contains useful signal, but shared errors can survive a consensus.

Available scored predictions have unequal configuration coverage. The average covers 100 public tasks; the oncology view is one task with eleven configurations. Correlations are not percentages correct. Neither view measures human scientific ability.

Sources and evidence

Sources

  1. Science Prediction Universe, Appendix E.8
  2. UniPat AI: results and scored coverage

Supporting facts

  • In the study’s 100 public tasks, mean pairwise agreement between scored configurations was a Spearman rank correlation of 0.52, versus a mean correlation of 0.33 with measured outcomes; agreement was higher on 81 tasks. [1]
  • Averaging item positions into a consensus ranking raised the score by 0.081 over the average submission, with a 95% confidence interval of 0.054 to 0.110, and improved results on 73 of the 100 public tasks. [1]
  • Agreement contained useful signal in SPU, but could be jointly wrong: across tasks agreement and accuracy had a Spearman correlation of 0.56, while eleven configurations agreed at 0.90 on one oncology task and averaged 0.27 against measurements. [1]
  • The authors interpret shared errors as agents relying on simple assumptions about how outcomes rise or fall; averaging can cancel errors that differ between configurations while retaining their shared errors. [1]
  • Scored coverage was unequal: Claude Opus 5.5 and Claude Fable 5.1 were scored on 44 and 30 public tasks respectively, largely because of safety refusals, and only 28 tasks across the full benchmark were scored by all 13 configurations. [1] [2]
  • The reported evaluation provides no human prediction baseline, so it does not establish whether these configurations outperform scientists, and its specified ranking tasks do not measure every kind of AI-assisted science. [1]

Method

Reported public-task agreement and measurement correlations share a -1 to 1 axis. Dot x = left + (correlation + 1) / 2 × axis width. Pooling paths are labelled schematic. Consensus gain is 0.081 score points (95% CI 0.054 to 0.110), higher on 73/100 public tasks.

Scope

Available scored predictions have unequal configuration coverage. The average covers 100 public tasks; the oncology view is one task with eleven configurations. Correlations are not percentages correct. Neither view measures human scientific ability.

Infographic by CryptoSlate

The authors connect those shared errors to simple assumptions about how experimental outcomes should rise or fall. If several agents rely on the same mistaken relationship, collecting more of their opinions can preserve that error.

Related Reading

Anthropic’s new AI leads freelance benchmarks, yet fails 58% of the tasks

The configurations were scored on different task sets. Claude Opus 5.5 and Claude Fable 5.1 had scores on only 44 and 30 public tasks, respectively, largely because of safety refusals. Across the full benchmark, all 13 configurations shared just 28 scored tasks: 27 public and one private.

The public tasks also looked backward at published measurements and were selected after pilot evaluations. The private set contained only 16 tasks across four unpublished experimental series, limiting how broadly its findings can be applied.

There was useful predictive ability in that private set: twelve of the 13 configurations ranked the unpublished experiments significantly better than chance overall, according to UniPat’s Nov. 12 results release. Of 208 official private runs, 159 were scored.

The study used no human prediction baseline. Its findings apply to ranking the experimental outcomes it tested, with lab measurements still needed to check the answers.

Related Reading

AI agents just completed a four-day cryptography test with only 20 human messages needed – DeepMind says

Featured AI Research

Editorial credits

Liam 'Akiba' Wright Liam ‘Akiba’ Wright Editor-in-Chief • CryptoSlate

Also known as “Akiba,” Liam Wright is a reporter, podcast producer, and Editor-in-Chief at CryptoSlate. He believes that decentralized technology has the potential to make…

X LinkedIn More from Liam

A robot arm inserts a charger beside code and a label showing 221 successful trials out of 240.

AI

Robots now run AI-written code without an AI calling the shots, succeeding 92% of the time

2 min read 17 hours ago

More coverage

AI helped hackers gain access at 10 unnamed organizations, researchers say

Nov 15, 2026

Microsoft found a way to turn a helpful AI into a sleeper agent

Nov 14, 2026

Coinbase AI trading tool pauses bots, but leaves orders open

Nov 14, 2026 Bearish

This article originally appeared on CryptoSlate. Read the full article at the source: https://cryptoslate.com/ai-models-agreed-with-each-other-more-than-with-reality-in-a-new-test/

More from Blockchain News

Leave a Reply

Your email address will not be published. Required fields are marked *