Ateliers · 03
Essai assay
Reproducible evaluation of models on scientific tasks.
An essai is the assay: the test that establishes what a casting actually contains.
Intended scope
- Task definitions as versioned files: inputs, references, metric, and the exact prompt or scoring procedure.
- Runs against a model pinned to a commit hash, with fixed seeds and decoding settings, producing a run record and per-item outputs.
- Checks for overlap between evaluation items and a training manifest from Affinage.
- Comparison across models, revisions and Burin edits, reported with uncertainty intervals rather than single numbers.
Not in scope
A public leaderboard; evaluation through hosted model APIs unless a task definition requires it explicitly.
First milestones
- Task-file format and validator.
- Deterministic evaluation of a tiny local model on a synthetic task, inside the test suite.
- Result comparison with bootstrap intervals.