Benchmarking AI research taste against human publications

The problem

AI scientists can now carry out much of the research process on their own, from generating ideas and grounding them in prior work to running experiments and writing papers according to arxiv.org. Frontier laboratories deploy them at growing scale, and their work has begun to enter the scientific literature according to arxiv.org. Yet executing research well matters only when the research itself is worth doing, a choice that rests on research taste according to arxiv.org. Research taste is the judgment of what to read, which problems to pursue, and which methods to try, and it increasingly shapes the work of AI scientists according to arxiv.org. Despite its growing importance, research taste remains far less understood than the ability of AI to execute research according to arxiv.org. Evaluating research taste has been challenging because existing work mostly judges one idea at a time through expert ratings, fixed benchmarks, or model judges, or measures how much an AI population repeats itself rather than whether its choices resemble those of human researchers according to arxiv.org.

The idea

The authors introduce IdeaTaste, a benchmark that scores a language model by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026 according to arxiv.org. The experiments span six contemporary language models and four research areas ranging from established fields to emerging fields, covering more than 90,000 proposals in total according to arxiv.org. The four research areas include machine learning systems, code generation, world models, and automated research according to arxiv.org. The benchmark evaluates proposals at two levels: the literature they cite, which reflects what they read, and the ideas they state, which reflects which problems they pursue and what methods they propose according to arxiv.org. At each level, matching measures agreement with human papers and diversity measures how varied the proposals are according to arxiv.org.

The results

According to the authors, language model agents fall well short of human communities in both the literature they cite and the ideas they state according to arxiv.org. Citation matching reaches about 26 percent of the human level, and idea matching reaches only 18 percent according to arxiv.org. No population reaches half of the human level on either metric according to arxiv.org. Populations of models can cite as diversely as a human community and take up its tasks and methods, but they rarely arrive at the concrete approaches and research plans the human community pursues according to arxiv.org. Neither richer field context nor larger populations substantially close this gap according to arxiv.org. In blinded expert review, AI proposals trail 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches according to arxiv.org. Proposals whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities according to arxiv.org.

Why it matters

These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue according to arxiv.org. To support ongoing evaluation, the authors release IdeaTaste as an open, continuously updated leaderboard that benchmarks contemporary AI proposals against future publication cycles according to arxiv.org.

Caveats

According to the authors, citation lists are an imperfect standard for measuring research relationships according to arxiv.org. The authors also note that human review can be inconsistent according to arxiv.org.