Contributor guide

From proposal to merged task

Every task goes through the same five stages, and each task page shows where it stands. This guide covers what happens at each stage, what you need to do, and what a finished task must meet to be accepted.

The five stages

A proposal is a short scientific description. A task is the runnable version of it in the benchmark repository: an environment, instructions for the agent, a reference solution and an independent verifier. The task board tracks each proposal through these stages:

Proposal submittedYou describe the problem with the proposal form. It becomes a GitHub Discussion and a task page.
Scientific reviewA domain reviewer approves it, asks for changes, or declines it.
Task implementationYou build the task and open a PR in the benchmark repository.
Agent evaluationMaintainers run the oracle, a negative control and frontier agents.
Accepted & releasedThe verified task joins a versioned release.

Three places, one rule each: this website is where you write and revise your proposal and follow its progress; the GitHub Discussion is the structured record that review and automation read; Discord is where questions and conversation happen.

01 Propose

Use the proposal form. It has four short sections (the scientific problem, the environment, the evaluation, and the contributors). Sign in with GitHub to submit.

What makes a proposal ready for review

Scientific value. Researchers in your field would consider the problem worth solving on its own merits, and its difficulty comes from the science itself.
Verifiable. Correctness can be judged by a script, a metric with a threshold, or an explicit procedure. Say what counts as a pass.
Frontier difficulty. Current agents should find it genuinely hard, or the workflow itself should be meaningful scientific work.
Sources, where they exist. If the task builds on papers, datasets or code, citing them (authors and year, a DOI, an arXiv ID or a link) helps reviewers check the science.
No easy leakage. Explain why the answer cannot simply be looked up, and what stays hidden from the agent.
Within budget. The current limit is one GPU and 16 CPU cores per task.

As you fill in the form, a proposal checklist suggests citing sources, points out missing pass criteria, compute over the limit, math that will not render, and proposals with a near-identical title. It is advice only, and never stops you from submitting.

Markdown and LaTeX work in every text field ( $…$ inline, $$…$$ for display math). After you submit, you can revise the proposal at any time with Edit proposal on its task page. Saving updates the page and the Discussion together.

02 Scientific review

A reviewer from your domain reads the proposal and publishes a structured review. It appears as a reply in the GitHub Discussion and on the task page.

What the reviewer records

Decision: Approved, changes requested, or rejected.
Short description and tags: How the task is summarised and filed on the board.
Difficulty: For example Moderate, Hard or Expert.
Scientific value: Why the problem matters to the field.
Primary metric and verification method: How success is measured and how it is checked. These become the core of your verifier.
Optional estimates: Runtime, compute and token budgets, baseline results and expected failure modes.

The three outcomes

Approved

Move on to stage 3 and build the task. The review's metric and verification method are your specification.

Changes requested

Revise with Edit proposal on your task page. The reviewer sees the updated Discussion.

Rejected

The proposal is not a fit in its current form. The review explains why. A substantially different proposal is welcome.

Get email when your review arrives. Open your proposal's GitHub Discussion (the task page links it) and press Subscribe in its sidebar. GitHub then emails you about every reply, including the review.

03 Build the task

Tasks are native Harbor task directories, following the Terminal-Bench Science contribution model. Each task lives in its own folder:

tasks/<domain>/<field>/<task-slug>/
├── task.toml
├── instruction.md
├── environment/Dockerfile
├── solution/solve.sh
└── tests/test.sh

task.tomlMetadata, resources and timeouts. The PR check requires a [metadata] table, and a name under [task] in org/name form, such as ai4sbench/<task-slug>.

instruction.mdThe only thing the agent reads. State the task, the inputs, where outputs go and what they must contain, in plain words. Never include the answer.

environment/DockerfileThe agent's environment. Pin versions, and keep reference answers out of the image.

solution/solve.shThe oracle: a reference solution that passes the verifier. It proves the task is solvable.

tests/test.shThe verifier. It checks the agent's outputs independently of anything the agent controls, and gives the same result on every run.

The <task-slug> is lowercase letters, digits and hyphens. Shell scripts start with a shebang (#!/usr/bin/env bash) and use Unix (LF) line endings. Windows line endings break them inside the Linux container.

A task.toml to start from

For a new task, the first selected proposal domain is primary. Use its folder name: materials-science, physics, chemistry, biology, ai-ml, applied-mathematics, or interdisciplinary. Set [metadata].domain and [metadata].field to exactly match the two folders. Existing tasks may retain their older paths and metadata when repaired.

These are the fields Terminal-Bench Science tasks carry. The benchmark repository's own example task shows the minimal shape the PR check accepts.

schema_version = "1.0"

[task]
name = "ai4sbench/<task-slug>"
description = "One sentence: what the agent must do."

[metadata]
author_name = "Your Name"
author_email = "[email protected]"
author_organization = "Your institution"
domain = "applied-mathematics"        # matches the <domain> folder
field = "numerical-analysis"         # matches the <field> folder
subfield = "numerical PDE solvers"
tags = ["multigrid", "python"]
expert_time_estimate_hours = 5

[agent]
timeout_sec = 7200.0

[verifier]
timeout_sec = 1200.0

[environment]
cpus = 4          # at most 16
memory_mb = 4096
gpus = 0          # at most 1

Run it before you open the PR

The PR template asks you to confirm that you ran the oracle locally. Maintainers run tasks with Harbor, pinned to a specific release. The benchmark repository's bench/README.md documents the pinned version and the runner. At a minimum, check that the oracle passes your verifier and that a do-nothing attempt fails it.

03 Open the task PR

Fork AI4S-Bench/ai4s-benchmark and create a branch.
Add exactly one task folder under tasks/<domain>/<field>/<task-slug>/.
Run the pre-flight check on your folder. It reads the files in your browser and uploads nothing.
Open the PR. In the template, paste your approved proposal's Discussion link under “Task Proposal”, summarise the research workflow, and tick the checklist.

The repository's validate-task-contribution check then runs on your PR. It requires the five files, and a task.toml with a [metadata] table and an org/name [task].name. For a new task, the PR description must also link its proposal Discussion in the benchmark repository; a repair to an existing task does not need a new proposal. The pre-flight check runs the same rules and can also check an open PR by its link.

Open the pre-flight check

04–05 Evaluation & release

Once maintainers link your PR to the proposal, the task page shows it in the timeline, and evaluation begins after the merge. The current validation matrix runs the oracle several times (it must pass every time), a do-nothing agent as a negative control (it must fail), and frontier agents to calibrate difficulty and record how they fail. Results appear in the task page's Agent results section.

Verified tasks join a versioned release. A task accepted into AI4S-Benchmark earns its authors contribution credit, visible under their names permanently. Task proposals for the ICML 2027 submission close on 31 December 2026.

Acceptance checklist

What a finished task must meet before it is accepted:

The proposal is approved, and the PR links its Discussion.
The PR adds one task at tasks/<domain>/<field>/<task-slug>/ with all five files, and task.toml has [metadata] and an org/name [task].name.
instruction.md is self-contained, states the expected outputs, and does not reveal the answer.
The oracle passes the verifier on every run, and a do-nothing attempt fails it.
The verifier judges outputs independently of agent-controlled code, as the review's verification method describes.
The reference data and answers are not reachable from the agent's environment.
Resources stay within one GPU and 16 CPU cores, and agent and verifier timeouts are set.
The shell scripts have a shebang and LF line endings, and the pre-flight check passes.

Questions at any stage? Ask on Discord , or read the repository's CONTRIBUTING.md .