Quality judges

What a quality judge is#

A translation engine writes translations. A quality judge never does: it reads a source segment and its translation, answers a fixed set of questions about the pair, and Localization OS turns the answers into a score from 0 to 100 and a verdict for a workflow step's QE gate. Because a judge never translates, it appears only in a step's QE judge engine picker. Every translation picker (project default, language pair, step provider and fallback, install default) hides it, and saving one there is refused.

Localization OS speaks the Jev System One protocol. Jev, by TypeSafe, is a typed judge: for each segment it answers five questions, accuracy, fluency, terminology and style, each as a level from none through minor and major to critical, plus a yes-or-no "did the meaning change", with a probability for every answer. A Jev-compatible server is any program that answers the same requests, so you can run the judge on your own hardware instead of calling the hosted service.

Hosted Jev#

1. Open Setup AI and scroll to Engines. Judges are listed in their own group, Quality judges.
2. Open + Add engine, give it a name, choose the kind Jev (TypeSafe, hosted), paste your TypeSafe API key and press Add engine. The address is fixed to https://api.typesafe.ai and cannot be changed, so the key is only ever sent to TypeSafe.
3. On the new card, open Edit and press Test. It lists the models the service offers, says whether the configured model is among them, and asks one small question, reporting the latency of each call.

If the install already has a TYPESAFE_API_KEY environment variable and no hosted Jev engine exists yet, an administrator sees a Create a Jev engine from TYPESAFE_API_KEY button in the Quality judges group. Pressing it moves the key into an encrypted engine that never shows the key again, and pressing it a second time creates nothing new.

Every hosted call is priced. The spend cap is checked before each hosted request, and the reported usage is recorded after the response.

Run SemIf on a GPU machine#

SemIf reads typed answers directly out of an open-weight language model. semif-serve is a small HTTP server that puts SemIf behind the Jev protocol, so Localization OS talks to it exactly as it talks to hosted Jev, only at a different address and for free. Both are third-party projects (see Credits and licences): Localization OS does not ship, bundle or install them or any model code, and semif-serve's own README is the authority on running it. The steps below are the path we have used.

What you need

• An NVIDIA GPU with about 10 GB of free video memory.
• git and uv.
• Disk space and bandwidth for the model: the first start downloads the model semif-serve is pinned to, several gigabytes.

Install

git -c url."https://github.com/".insteadOf="git@github.com:" clone --recurse-submodules https://github.com/dddanielliu/semif-serve.git
cd semif-serve
uv sync
uv run --no-sync python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

semif-serve pulls SemIf in as a git submodule addressed over SSH; the -c url...insteadOf part fetches it over HTTPS instead, so no GitHub SSH key is needed.

The last command must print True. On Windows, uv sync installs the CPU-only build of torch, so swap in the CUDA build at the version semif-serve pins, then run the check again. It should print a version ending in +cu128 and True:

uv pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128

2.10.0 was the pinned version when this page was written; match whatever semif-serve's lock file pins. If the check still reports a CPU build, repeat the install with --reinstall.

SemIf can optionally use faster attention kernels (uv pip install flash-linear-attention causal-conv1d). Without them it logs a warning and uses a slower reference path. On Windows these packages often cannot be installed, and the slower path is normal there.

Start the server

# Linux or macOS shell
CUDA_VISIBLE_DEVICES=0 PYTORCH_ALLOC_CONF=expandable_segments:True SEMIF_PORT=8077 uv run --no-sync semif-serve

# PowerShell on Windows
$env:CUDA_VISIBLE_DEVICES = "0"
$env:PYTORCH_ALLOC_CONF = "expandable_segments:True"
$env:SEMIF_PORT = "8077"
uv run --no-sync semif-serve

PYTORCH_ALLOC_CONF=expandable_segments:True lets PyTorch use video memory more flexibly. semif-serve's README requires it on a 12 GB card, where the model can otherwise run out of memory partway through a batch, and it does no harm on larger cards.

Always start it with uv run --no-sync. A plain uv run re-syncs the environment, which can put the CPU build of torch back, and the server then runs very slowly on the CPU.

The settings it reads:

Check that it answers, from a second terminal on the same machine:

curl http://127.0.0.1:8077/v1/models

Connect it on the same machine#

When SemIf runs on the same machine as Localization OS:

1. Open Setup AI, then + Add engine under Engines.
2. Name it, for example "SemIf (this machine)", and choose the kind Jev-compatible local server (SemIf, Kev).
3. Set the Base URL to http://127.0.0.1:8077. Leave the API key blank unless you set SEMIF_API_KEY, and leave Model blank, which means jev-latest. The confidence floor, meaning-changed floor and timeout are optional: blank floors use the install defaults, and a blank timeout uses the fixed default (30 seconds for a local server, 10 seconds for hosted Jev).
4. Press Add engine, then on its card open Edit and press Test. You want "Jev reachable", a model list, and "one-question check OK".
5. Press Inspect on the card and use Try it with a translation you know is wrong, to watch the judge catch it (see Inspect a judge and try it).
6. Pick the engine as a workflow step's QE judge engine (see Use a judge in a workflow).

From the command line:

loc-tms engine create --name "SemIf (this machine)" --kind jev_local --base-url http://127.0.0.1:8077

Connect it over Tailscale#

Use this when the GPU is in a different machine from Localization OS, for example a GPU workstation serving an install that runs elsewhere. Tailscale gives both machines private addresses on your own network, called a tailnet, and encrypts the traffic between them.

On the GPU machine

1. Install Tailscale and sign in, so the machine joins your tailnet. tailscale ip -4 prints its tailnet address, a 100.x.y.z address, and the Tailscale admin console shows its MagicDNS name.
2. Start semif-serve listening on that tailnet address only, with a key. Use the exact address tailscale ip -4 printed. Do not bind it to every interface, which would also expose it on your office or home network.

# Linux or macOS shell
CUDA_VISIBLE_DEVICES=0 PYTORCH_ALLOC_CONF=expandable_segments:True \
  SEMIF_HOST=100.x.y.z SEMIF_PORT=8077 SEMIF_API_KEY=a-long-random-value uv run --no-sync semif-serve

# PowerShell on Windows
$env:CUDA_VISIBLE_DEVICES = "0"
$env:PYTORCH_ALLOC_CONF = "expandable_segments:True"
$env:SEMIF_HOST = "100.x.y.z"
$env:SEMIF_PORT = "8077"
$env:SEMIF_API_KEY = "a-long-random-value"
uv run --no-sync semif-serve

The key is a password, not encryption. semif-serve speaks plain HTTP, so your text and the key are protected in transit only because Tailscale encrypts everything on the tailnet. Keep the server on its tailnet address and never forward its port to the internet. A firewall rule on the GPU machine that allows the port only from the Tailscale interface is a sensible extra layer. Start semif-serve after Tailscale is connected: binding to the tailnet address fails while that address does not exist yet.

In Localization OS

1. In Setup AI, open + Add engine and choose the kind Jev-compatible local server (SemIf, Kev).
2. Set the Base URL to the machine's MagicDNS name or its tailnet IPv4 address, with the port, for example http://100.x.y.z:8077.
3. Paste the same value you gave SEMIF_API_KEY as the engine's API key. Whenever the server was started with a key, the matching key is required: without it every request fails authentication. semif-serve refuses to listen anywhere but loopback without a key, so only a server started without one, on the same machine as Localization OS, can be used keyless.
4. Press Test, then pick the engine as a step's QE judge engine.

The machine that runs Localization OS must itself be able to reach the GPU machine's tailnet address, which normally means Tailscale is installed and signed in there too. If Localization OS cannot reach the address at all, no URL will make it work.

Use a judge in a workflow#

1. Open the workflow and edit the Translate step you want gated.
2. Under Options, tick QE routing gate, set the QE threshold (0 to 100, default 70), and pick the judge as the QE judge engine. Save.
3. Optionally tick Auto-fix failures, which appears only when the QE judge is a Jev judge, so failing segments get a bounded rewrite before they reach a person.

A segment that scores at or above the threshold with every dimension answered confidently passes. One below the threshold fails and goes to review, or first to the auto-fix post-editor when auto-fix is on and no dimension is unsure. One that scores high enough but where the judge was unsure about a dimension is undecided: it carries on unchanged, is counted as QE skipped, and shows a QE undecided badge in the editor rather than going to review. Add a Review step if those segments must be seen by a person. The Workflows page covers the gate and auto-fix in full.

Inspect a judge and try it#

Every judge card in Setup AI has an Inspect button, available to administrators. The page has four parts:

1. What Jev is asked: the five questions, what every level means, and the exact request body. The API key travels only in an authorization header and is never shown.
2. How it decides: the confidence floor, the meaning check, how an unsure answer is handled, the score formula and the pass threshold. It is rendered from the same numbers the QE gate uses and from this engine's own settings, so it always matches what the gate does.
3. Try it: enter a source, a translation, both locales and a threshold, and one segment is scored exactly as a QE-gated step would score it. You see the request sent, the raw answer, each dimension's level, the verdict (pass, fail, undecided or not scored), the cost and latency, and the exact prompt the auto-fix post-editor would receive for its first round.
4. Ledger: a link to the LLM Ledger filtered to this engine. Tries are recorded there while the Ledger is on.

A try on hosted Jev is billed like any other call and stops with a clear message once the spend cap is reached; a try on a server you run is free. Try it is rate limited, and hosted requests are spend-checked.

Evaluate any judge, hosted or one you run, on representative segments and language pairs of your own before you rely on it. Inspect, then Try it, is the way to do that: feed it real segments from your projects, including ones you know are wrong.

Command line, API and MCP#

Everything above is also available headlessly. Engine ids below are examples; loc-tms engine list shows yours.

loc-tms engine create --name "Jev" --kind jev --secret api_key=YOUR_TYPESAFE_KEY
loc-tms engine create --name "SemIf" --kind jev_local --base-url http://127.0.0.1:8077 \
  --config confidence_floor=0.6 --config meaning_floor=0.8 --config timeout_s=30
loc-tms engine import-jev-env
loc-tms engine test 7
loc-tms engine inspect 7
loc-tms engine jev-try 7 --source "Pay ten euros." --target "Zahlen Sie neun Euro." \
  --source-locale en --target-locale de
loc-tms workflow edit-step 12 3 --qe-gate --qe-provider engine:7 --qe-autofix

The --config values are optional: unset floors use the install defaults, and an unset timeout uses the fixed default (30 seconds local, 10 seconds hosted). loc-tms lqa jev-refine --judge engine:7 runs the same score-and-rewrite loop as an experiment over pairs you supply; it writes no production content.

Over the API, a judge is created like any engine with POST /api/admin/engines and changed with POST /api/admin/engines/{id}/update. Its key travels only over those admin API routes, never over MCP. GET /api/admin/engines/{id}/inspect and POST /api/admin/engines/{id}/jev-try are the Inspect page's two halves, and POST /api/admin/engines/import-jev-env is the one-click import. The MCP server's install-management tool offers inspect_engine and jev_try, and never carries a key.

Troubleshooting#

Credits and licences#

Localization OS talks to these judges over the network. It does not ship, bundle, install or redistribute any of them, or any model weights. They are third-party projects with their own terms, outside the Localization OS licence and support, so evaluate each one yourself before you use it in production.

• Jev and the System One protocol are TypeSafe's. Hosted Jev is used with your own TypeSafe account and key, under TypeSafe's terms.
• SemIf (github.com/TheoLeeCJ/SemIf) is the library that reads typed answers from an open-weight model. It is published under the MIT licence.
• semif-serve (github.com/dddanielliu/semif-serve) is the HTTP server that puts SemIf behind the Jev protocol. At the time of writing its repository has no licence file, so its terms of use are unstated; decide for yourself whether that is acceptable for your use.
• Kev (github.com/jaredpalmer/kev) is a family of Jev-style models that speaks the same protocol, published under the Apache 2.0 licence. Add it exactly like SemIf, as a Jev-compatible local server at its own address and port. It needs more video memory than SemIf; check its README before sharing a card with another model.

SemIf, semif-serve and Kev are third-party projects; SemIf's own repository states that it is independent and not affiliated with TypeSafe. SemIf and Kev carry the licences named above; semif-serve has no licence file. Thanks to the authors of all three for making a local judge possible.

Current limits#