🔍 Read the full analysis: Should You Trust AI To Judge Research Novelty? on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Ai2 researchers report that six large language models gave inconsistent judgments about the novelty of scientific ideas across different evaluation setups. In one pairwise test, changing a few prompt words shifted accuracy by more than 50 percentage points; retrieval and additional reasoning did not reliably improve results. The team released a benchmark, data, code and an evaluation pipeline, while warning that some benchmark labels are imperfect.
Ai2 researchers report that six large language models gave unstable novelty judgments when researchers changed how the ideas were presented or evaluated, as detailed in the original analysis. In one pairwise test, a few prompt-word changes shifted accuracy by more than 50 percentage points, raising questions about using model scores to screen research proposals before human review.
The study tested GPT-5.1, GPT-5.2 and GPT-5.4, as well as Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. Researchers asked the models to rate novelty directly and to compare ideas in pairs. The authors report that adding literature retrieval or allowing more reasoning did not consistently improve judgments. They also say two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
For one benchmark setting, the team used ICLR 2026 peer reviews to find explicit comments on originality. Its high-novelty group includes 154 ideas from accepted papers in the top-rated 10% of their areas, selected when reviewers praised novelty and none disputed it. Its lower-novelty group includes 145 ideas from rejected papers in the bottom 10%, selected when reviewers raised originality concerns and none praised novelty.
A human annotator agreed with the direction of the extracted novelty signal in 47 of 50 checked cases; the authors say the three disagreements did not change the review classifications. A second setting compares the 154 human-authored ideas with 154 proposals from a basic language-model generator. The researchers treated the generated ideas as less novel but describe those labels as weak, since each idea was not individually verified. Ai2 has released the paper, benchmark data, code and an automated pipeline for building additional evaluation sets.
Unstable Scores Complicate Idea Screening
Research groups may use language models to filter proposals before committing time to literature searches, experiments or expert assessment. If a model’s verdict changes substantially after small prompt edits, its score may reflect evaluation choices as well as the idea itself. That makes it harder to compare proposals across teams or to treat a single score as dependable evidence of originality.
The findings also matter to projects that generate hypotheses or synthesize research. Such systems need credible ways to check whether a proposal adds something not already known. The study does not establish that all AI novelty assessments fail; it indicates that performance on one chosen setup is not, by itself, evidence of reliable judgment. Researchers and users will need to examine how results hold up under different prompts, models and comparison methods.
Products Worth Considering
DJI Osmo Mobile 8 3-Axis Smartphone Gimbal Stabilizer
DJI Osmo Mobile 8 Gimbal with Mic Mini Transmitter, Infinity Black
Bose SoundLink Flex Bluetooth Speaker (2nd Gen), Portable Waterproof Speaker, Black
As an affiliate, we earn on qualifying purchases.
How Ai2 Tested Novelty Judgments
A novelty judge is a model asked to estimate whether a research idea adds something new relative to prior work, either by rating it alone or comparing it with another proposal. The Ai2 authors say there is no standard recipe for these evaluations: teams can choose different models, prompts, reasoning budgets, retrieval systems and formats. Those choices can make results difficult to compare.
The authors also argue that evaluations often rely on human-written papers even when a tool is meant to screen AI-generated ideas. Their benchmark includes separate settings for ideas with reviewer evidence about originality and for human-authored ideas compared with generated proposals. The team cautions that paper acceptance or rejection alone is not a sound novelty label, because decisions can depend on factors other than originality.
Reviewer comments provide evidence about how reviewers perceived novelty, not a definitive measurement of whether an idea is original across all prior research. The study’s benchmark is therefore a test resource rather than a final standard for judging scientific contribution.
“Seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.”
— Ai2 study authors
Limits of the Benchmark Evidence
The benchmark’s labels have limits. Reviewer comments and selection criteria indicate perceived originality, but they do not establish a definitive novelty ground truth. In the human-versus-generated setting, the team assumed the generated ideas were less novel; those labels were not checked individually. An expert assessment covered 30 pairs, selected heavily from cases the strongest judge got wrong. The authors report that the expert preferred the human idea in every pair and judged none of the generated ideas novel, but that small check does not validate the full set.
The supplied study description does not provide results for every model, prompt, reasoning budget or evaluation format. It also does not show whether the reported patterns hold in other disciplines, with different idea generators, or in systems with broader access to research literature. The materials do not specify the release date or announce a planned follow-up study. The extent to which the instability persists across all evaluation settings remains unclear.
Further Tests Across Research Fields
The released benchmark, data, code and pipeline give other researchers resources to test the findings and build additional evaluation sets. Further work can examine whether prompt sensitivity appears across disciplines, idea sources and judge designs, and whether particular combinations of retrieval or model methods yield more consistent results. Ai2 has not announced a specific follow-up timeline in the supplied material, so no next release or milestone is confirmed.
For now, the study supports treating novelty scores as evaluation-dependent signals rather than definitive verdicts. A practical next step for research teams is to check how their chosen system behaves across more than one prompt or evaluation format and to keep expert review in the process. Whether any approach can produce dependable novelty judgments across settings remains an open research question.
Key Questions
What did the Ai2 study find?
The researchers report that six language models gave unstable judgments of scientific idea novelty across evaluation setups. In one pairwise experiment, changing a few prompt words shifted accuracy by more than 50 percentage points.
Did literature retrieval or more reasoning make the models more reliable?
Not consistently, according to the authors. Their study reports that retrieval and additional reasoning did not reliably improve judgments across the tests described.
Does the study prove AI cannot judge research novelty?
No. It reports instability in the tested models and setups; it does not establish that all novelty evaluations fail. The authors’ results also depend partly on benchmark labels drawn from reviewer comments and selection criteria, which are not definitive measures of originality.
What did Ai2 release?
Ai2 released the paper, benchmark data, code and an automated pipeline intended to help researchers build and test additional evaluation sets.
What remains unknown?
The supplied material does not establish whether the results generalize across disciplines, other idea generators or systems with broader literature access. It also does not identify a specific follow-up study or timeline.
Primary source: Hugging Face · via ThorstenMeyerAI.com
