Where are the questions?
Questions drive discovery, and today they also drive the progress of AI agents. In interdisciplinary fields especially, an agent can assemble strong solutions from a broad literature, yet the questions that matter most still live with domain researchers. AI4S-Benchmark is a community effort to collect those questions, together with the researchers who know them best.
Why now?
AI systems are becoming capable of carrying out increasingly complex engineering and research workflows, but their true scientific capabilities are still poorly characterized. Coding has mature executable evaluations; many scientific domains do not. As agents begin to take part in real research, the lack of rigorous measurement becomes the bottleneck.
Genuinely valuable scientific problems cannot simply be scraped from the web. Scientific software, data and evaluation expertise are scattered across laboratories, and determining whether an output is scientifically correct often requires domain expertise.
Researchers working at the frontier know which problems actually matter, which workflows are genuinely difficult, where current agents hallucinate or fail, what constitutes scientifically meaningful progress, and how a result can be properly verified. AI4S-Benchmark turns that distributed expertise into open, executable scientific evaluations.
Why interdisciplinary science?
Important scientific discoveries frequently emerge at disciplinary boundaries, while successful AI agents increasingly need to combine knowledge, reasoning, code, tools and scientific domain expertise. Interdisciplinary tasks are therefore especially valuable for measuring emerging scientific capability, and this benchmark actively encourages them.
What counts as a good task?
Every task must clear three bars:
- Scientific value. Researchers in the field would consider the problem worth solving on its own merits. Difficulty invented purely to confuse an AI does not qualify.
- Verifiable. Correctness can be judged through a script, metric, experimental criterion or explicit scientific evaluation procedure.
- Frontier difficulty. The task remains genuinely challenging for current agents. Open problems qualify; artificial difficulty and trick questions do not.
Each task also states its integrity measures up front: the contamination risk, possible routes for answer leakage, and the anti-cheating measures the environment enforces.
How are tasks reviewed?
Every task follows the same transparent path, recorded on GitHub:
- Submit. A researcher proposes a task through the proposal form.
- Scientific review. Automated checks verify format and duplication; domain reviewers assess scientific value, verifiability and difficulty.
- Revision. Authors revise their proposal on its task page, and the review record updates with it. Discussion happens on Discord ; the GitHub Discussion holds the structured record that review and automation read.
- Task PR. The specification, environment, evaluator and anti-cheating measures are submitted as a structured pull request.
- Agent testing. Frontier agents are run; pass rate, runtime, cost and failure modes are recorded.
- Release. Verified tasks enter a versioned release.
The task board shows where every task stands: Pending Review, Changes Requested, Approved or Rejected during review, then Task PR, Agent Testing and Released. The contributor guide walks through each stage.
How are agents evaluated?
Agents run inside the task's execution environment under a recorded configuration: harness version, model and scaffold identity, budgets and limits. Scoring is performed by the task's evaluator, never by code the agent controls, against reference data that stays hidden from the agent wherever the task design requires it.
An agent scaffold and its underlying foundation model are tracked separately. Published results carry score, cost, runtime and a verification status; a result is marked Verified only when it was reproduced or validated under the AI4S-Benchmark evaluation procedure. Detailed methodology lives with the harness in the benchmark repository.
Who governs the project?
AI4S-Benchmark is community-governed. Contribution points, roles (Active Contributor, Core Contributor, Maintainer) and area committees are defined in the public credit and governance policy. Maintainers participate in task acceptance, review organization, contributor management and roadmap decisions. Scientific areas may form field-specific committees that organize reviews and decide task acceptance in their area.
What does open and independent mean?
Open means that everything happens in public on GitHub: task specifications, review history, evaluator code, releases and evaluation records. Independent means the benchmark is not owned by any single lab or model vendor, and no participant has private influence over task selection or scoring.
The long-term roadmap
The project moves through three phases:
- Bench (current): continuously test frontier AI agents on difficult scientific problems and track the moving frontier of AI capability in science.
- Dataset: accumulate tasks, execution environments, agent trajectories, evaluators and scientific verification systems into a scientific dataset of lasting value.
- Challenge: organize open challenges around important problems that remain unsolved, and revisit them as AI systems become stronger.
Detailed documentation, including schemas, validators, review guidelines and the evaluation harness, lives in the benchmark repository .