Essai · Guide
Guide
How a task file and a pinned model become a result that someone else can repeat, with its record and its uncertainty.
À venir This page is an outline. Each section below keeps its coming soon stamp until it describes code that exists in the repository.
Available now: install from source
There is no release yet. The scaffold installs from a clone of the repository:
git clone https://github.com/fonderie-dev/fonderie-essai.git
cd fonderie-essai
python -m pip install -e ".[dev]"
python -m pytest
The tests need neither a GPU nor downloads.
Planned chapters
Installation
Coming soonInstalling a released version, and the optional extras that bring in model libraries only when they are needed.
A first evaluation
Coming soonDeterministic evaluation of a tiny local model on a synthetic task, on CPU.
Writing a task
Coming soonTask definitions as versioned files: inputs, references, metric, and the exact prompt or scoring procedure.
Running an evaluation
Coming soonRuns against a model pinned to a commit hash, with fixed seeds and decoding settings.
Run record and per-item outputs
Coming soonWhat Essai writes for every run, following the Fonderie run record, and how to read the per-item outputs.
Train–test overlap
Coming soonChecking evaluation items against a training manifest from Affinage before a result is reported.
Comparing models
Coming soonComparison across models, revisions and Burin edits on the same tasks.
Reporting with uncertainty
Coming soonBootstrap intervals instead of single numbers, and how to read them.