Fonderie · Atelier 03

Essai assay

Reproducible evaluation of models on scientific tasks.

An essai is the assay: the test that establishes what a casting actually contains.

État Scaffold. The package installs and its smoke test passes; none of the scope below is implemented yet. This site is an outline of what Essai will document, and every detailed section is marked coming soon.

The outline GitHub ↗

Fonderie mark: a crucible pouring into an ingot mould whose cells read as a weight matrix

SommaireWhat this site will hold

PérimètreIntended scope

What Essai is meant to do. The scope is written before the code, so the code can be held to it; none of it is implemented yet.

  • Task definitions as versioned files: inputs, references, metric, and the exact prompt or scoring procedure.
  • Runs against a model pinned to a commit hash, with fixed seeds and decoding settings, producing a run record and per-item outputs.
  • Checks for overlap between evaluation items and a training manifest from Affinage.
  • Comparison across models, revisions and Burin edits, reported with uncertainty intervals rather than single numbers.

Not in scope

A public leaderboard; evaluation through hosted model APIs unless a task definition requires it explicitly.

JalonsFirst milestones

  1. Task-file format and validator.

    Coming soon
  2. Deterministic evaluation of a tiny local model on a synthetic task, inside the test suite.

    Coming soon
  3. Result comparison with bootstrap intervals.

    Coming soon

The journal →

La chaîneOne of three ateliers

  1. Affinage refining

    Training and fine-tuning open models on domain data, packaged as customised local models.

    Status: scaffold
  2. Burin engraving

    Inspecting and editing the weights of open models.

    Status: scaffold
  3. Essai assay

    Reproducible evaluation of models on scientific tasks.

    This site

    Status: scaffold

Every atelier follows the same rules for records, artefacts and safe loading: the Fonderie method ↗