Patois.

Docs · Features

Evaluations Lab

A sandbox for one question, which setup translates better, answered with statistics that refuse to declare a winner when the evidence is thin.

What it does#

The loop is always the same: dataset, candidates, run, compare, promote. A dataset is what you test on, pasted text or a frozen snapshot of a real document's segments. A candidate is one setup under test, usually a workflow; to compare two models, build two one-step workflows differing only in the model.

A run scores each candidate against the dataset, with a cost estimate you confirm before anything starts. Comparing candidates against a baseline returns a verdict per candidate, improvement, regression, tradeoff, or tie, with the reasoning behind it, never a bare number; below a minimum sample size the result is honestly labelled "indicative only" instead. Promotion is the one deliberate way out of the Lab: it turns a winning candidate into a real production workflow, and only when the statistics genuinely support the win at your chosen margin.

Getting started#

  1. Build a dataset: paste one segment per line, or snapshot a real document. Add a known-good translation per line to unlock the strongest metrics.
  2. Define what you are comparing: create a candidate pointing at a workflow.
  3. Launch a run: pick a dataset, candidates and scorers, then confirm the cost estimate.
  4. Compare results against a baseline.
  5. Promote the winner once the statistics support it.
eval dataset create --name "Support FAQ" --file ./lines.txt --target de-DE
eval candidate create --name "Qwen default" --workflow-id 4
eval run start --dataset 1 --candidate 4 --candidate 5 --budget 5.00 --yes
eval run compare --dataset 1 --baseline 4 --candidate 4 --candidate 5

Works with#

  • Workflows: a candidate is normally just a pointer at a workflow, and promotion turns the winner into a production workflow.
  • Documents: snapshotting a document copies its segments, and its current translations as references, into a frozen dataset.
  • Engines: candidates and the LLM judge both resolve engines; a local-engine experiment costs nothing.
  • Translation memory: a dataset snapshotted from a document inherits whatever translation memory already filled in, worth knowing before reading the scores.

Current limits#

Current limits

A metric that needs a reference translation is simply absent from the menu when the dataset has none; it is never shown as zero.

Below the minimum sample size, every result is labelled "indicative only," however large the difference looks.

If the judge model shares a family with a candidate being scored, the interface warns that it may favor itself; the deterministic metrics are the cross-check.

The cost estimate is a floor, not a promise; the budget ceiling set on the run is the real limit.

Editing a candidate that a run already used forks it rather than changing it in place, so past results stay meaningful.

On the compare screen, include the baseline in the candidate list too, or it silently compares against the wrong thing; this applies on the command line as well as in the web UI.

The Lab has its own switch: turned off, new runs and edits are refused, but existing results stay readable.