01
Submit
A researcher proposes a task through the proposal form: the problem, the environment, and how a result would be judged.
Open · Independent · Community-built
An interdisciplinary benchmark for AI agents in science.
Scientists contribute hard, open problems from their own fields. We turn them into verifiable tasks and measure how far frontier AI agents get.
Scientists bring the problems that matter. Agents attempt them. Every result is verified by domain experts, in the open, with permanent credit for the people who made it possible.
What we collect
Hard, open scientific problems.
Every task must clear three bars before it can enter a release.
Interdisciplinary tasks are especially encouraged.
Many of the most important questions sit at the boundaries between fields, where rigorous evaluation is rarest. An agent that combines knowledge, code and tools across disciplines demonstrates a scientific capability that single-domain question answering cannot reveal.
How it works
From proposal to release, in the open.
02
Scientific review
Automated checks screen it. Domain reviewers assess scientific value, verifiability and difficulty.
03
Task PR
The specification, environment, evaluator and anti-cheating measures are submitted as a structured pull request.
04
Agent testing
Frontier agents run the task. Pass rate, cost, runtime and failure modes are recorded.
05
Release
Verified tasks enter a versioned release, with every contributor credited.
Roadmap
Bench today. Dataset and Challenge next.
Three phases, with two publications planned along the way.
Updates
Latest from the benchmark
Community
Built by the people who know which problems matter.
The problems worth benchmarking are known best to the researchers working on them. AI4S-Benchmark turns that expertise into open, executable evaluations and gives permanent, visible credit for every contribution.