Essai · Guide

Guide

How a task file and a pinned model become a result that someone else can repeat, with its record and its uncertainty.

À venir This page is an outline. Each section below keeps its coming soon stamp until it describes code that exists in the repository.

Available now: install from source

There is no release yet. The scaffold installs from a clone of the repository:

git clone https://github.com/fonderie-dev/fonderie-essai.git
cd fonderie-essai
python -m pip install -e ".[dev]"
python -m pytest

The tests need neither a GPU nor downloads.

Planned chapters

  1. Installation

    Coming soon

    Installing a released version, and the optional extras that bring in model libraries only when they are needed.

  2. A first evaluation

    Coming soon

    Deterministic evaluation of a tiny local model on a synthetic task, on CPU.

  3. Writing a task

    Coming soon

    Task definitions as versioned files: inputs, references, metric, and the exact prompt or scoring procedure.

  4. Running an evaluation

    Coming soon

    Runs against a model pinned to a commit hash, with fixed seeds and decoding settings.

  5. Run record and per-item outputs

    Coming soon

    What Essai writes for every run, following the Fonderie run record, and how to read the per-item outputs.

  6. Train–test overlap

    Coming soon

    Checking evaluation items against a training manifest from Affinage before a result is reported.

  7. Comparing models

    Coming soon

    Comparison across models, revisions and Burin edits on the same tasks.

  8. Reporting with uncertainty

    Coming soon

    Bootstrap intervals instead of single numbers, and how to read them.