Fonderie · Atelier 03
Essai assay
Reproducible evaluation of models on scientific tasks.
An essai is the assay: the test that establishes what a casting actually contains.
État Scaffold. The package installs and its smoke test passes; none of the scope below is implemented yet. This site is an outline of what Essai will document, and every detailed section is marked coming soon.
SommaireWhat this site will hold
Guide
From a task file and a pinned model to a result with its record and its uncertainty.
- Installation
- A first evaluation
- Writing a task
- Running an evaluation
- Run record and per-item outputs
- Train–test overlap
- Comparing models
- Reporting with uncertainty
Référence
Every public function, file format and report, field by field.
- Python API
- Task-file format
- Run-record schema
- Per-item outputs
- Comparison report
Journal
What changed, when, and what was verified.
- Milestone notes
- Release notes
- Archived releases
PérimètreIntended scope
What Essai is meant to do. The scope is written before the code, so the code can be held to it; none of it is implemented yet.
- Task definitions as versioned files: inputs, references, metric, and the exact prompt or scoring procedure.
- Runs against a model pinned to a commit hash, with fixed seeds and decoding settings, producing a run record and per-item outputs.
- Checks for overlap between evaluation items and a training manifest from Affinage.
- Comparison across models, revisions and Burin edits, reported with uncertainty intervals rather than single numbers.
Not in scope
A public leaderboard; evaluation through hosted model APIs unless a task definition requires it explicitly.
JalonsFirst milestones
Task-file format and validator.
Coming soonDeterministic evaluation of a tiny local model on a synthetic task, inside the test suite.
Coming soonResult comparison with bootstrap intervals.
Coming soon
La chaîneOne of three ateliers
-
Affinage refining
Training and fine-tuning open models on domain data, packaged as customised local models.
Status: scaffold -
Burin engraving
Inspecting and editing the weights of open models.
Status: scaffold -
Essai assay
Reproducible evaluation of models on scientific tasks.
This site
Status: scaffold
Every atelier follows the same rules for records, artefacts and safe loading: the Fonderie method ↗