Databricks export

Enabling this feature ships your source content and translations out of the install to your Databricks workspace. That is the intended design: it is the training-data product, where your translators' post-edits land in your lakehouse as preference pairs. The feature is off by default, host-only (it can never be turned on over the API), read-only against every source table, and dormant until you deliberately turn it on. Nothing egresses without a deliberate, host-only operator action.

The exporter is a manual batch service. It stages per-stream artifacts locally, uploads them to a Unity Catalog Volume via the Databricks Files API, and, optionally, loads them into tables via COPY INTO on a SQL warehouse. No Spark, no Databricks Connect, no always-on daemon.

What ships: the six streams#

Customer-side setup: the entire ask#

Six configuration values, two of them secrets. Everything except the two secrets is a non-secret setting, set with the CLI or the Settings UI; the secrets are host-provided environment variables, never persisted, never logged.

1. Workspace URL, a bare workspace host (never an account-level URL).
2. SQL warehouse ID. A serverless warehouse is recommended; see cadence below.
3. Catalog name.
4. Schema name (you create the schema).
5. An OAuth client ID (service-principal app id), or a personal access token, as an environment variable.
6. An OAuth client secret, host-provided, never persisted, never logged.

A handful of other settings have sane defaults and are your call: which volume to use, whether to run COPY INTO automatically after staging (off by default, so you can load your own way), the export file format (newline-delimited JSON by default; Parquet is available but needs pip install pyarrow), which streams to export, and shard and deadline sizing for large batches.

Authentication. An OAuth machine-to-machine service principal is the supported path; the short-lived token is cached in memory and refreshed automatically before it expires, never persisted. Least privilege comes from the Unity Catalog grants below, not from the OAuth scope alone; the grant block is what actually confines the service principal. A personal access token is a documented legacy fallback using the same bearer-token header, though note that Databricks lets a workspace admin disable personal access tokens workspace-wide, and an unused one auto-revokes after a period of inactivity, so a token-only integration can be switched off by a policy you do not control. Rotating an OAuth secret is a supported, no-downtime environment swap.

The grant block: one schema, nothing account-level

-- One schema, one service principal. Nothing account-level.
GRANT USE CATALOG ON CATALOG <catalog>                          TO `<sp-app-id>`;
GRANT USE SCHEMA  ON SCHEMA  <catalog>.<schema>                 TO `<sp-app-id>`;
GRANT CREATE VOLUME, CREATE TABLE ON SCHEMA <catalog>.<schema>  TO `<sp-app-id>`;
-- after the one-time setup below creates the volume and tables:
GRANT READ VOLUME, WRITE VOLUME ON VOLUME <catalog>.<schema>.<volume> TO `<sp-app-id>`;
GRANT MODIFY, SELECT ON SCHEMA <catalog>.<schema>              TO `<sp-app-id>`;
-- plus: grant the service principal CAN USE on the SQL warehouse. This is a workspace
-- ACL, not a Unity Catalog grant, and may need the Permissions API/UI rather than SQL.
-- Do not omit MODIFY: it is required for COPY INTO to write the table.

Deliberately not asked for: schema or catalog creation (you create the schema yourself), any account-level or metastore-admin role, and any cluster or job permission. A one-schema blast radius, narrower than most connector-style integrations, is a genuine selling point.

One-time setup

After the grants above, run the one-time setup to create the volume and the six stream tables with explicit, versioned schemas, so you never write DDL by hand. It is idempotent, so re-running it only creates whatever is missing and leaves existing tables untouched.

loc-tms databricks setup
# or, against a running server: loc-tms databricks setup --remote <url> --token <key>

The same action is also available as POST /api/admin/databricks/setup, or as the "Create volume + tables" button on the admin page. It requires the connection already configured: credentials, workspace URL, and warehouse ID.

Content egress and privacy posture#

• Content leaves the install by design. Source text, target text, and the before and after post-edit text are the training data; enabling the flag ships them to your workspace. Your lakehouse is your own governed environment. The exporter scrubs known credential shapes out of every free-text column, but it does not scrub content otherwise; that is the point.
• Optional content PII scrub (off by default): a lossy, pattern-based scrub of personal-data shapes out of the content columns before egress. It can mangle legitimate translation content and is a convenience, not a compliance guarantee.
• Translator identity is pseudonymized by default. Each actor exports as a stable pseudonymous hash, derived from a once-generated, host-only salt. Rotating that salt forks identities in the lakehouse. Raw email is never exported unless you explicitly turn on a dedicated, host-only opt-in, and that opt-in is a one-way door: a single batch carrying raw emails permanently de-pseudonymizes the whole historical table, because everything joins on the same hash. Leave it off unless you mean it.
• Cross-customer mixing is the most important thing to get right in your own pipeline. Most rows carry a direct customer or organization id, but the translation-memory stream does not: its customer scope comes from a join through the project. Partition your downstream training by customer, joining the TM stream through its project first; a single model trained across customers is a cross-customer leak. This is required guidance for any training use of the export.

Cadence: batch on a schedule, never per-run#

A serverless SQL warehouse is recommended: it starts in a few seconds, while a classic warehouse takes minutes and bills a startup on every export. Run the exporter as a batch on a schedule, hourly or daily, never per translation run.

Snapshot growth. The current-state streams append a full dated snapshot each batch; there is no in-place overwrite yet. The whole-table snapshots grow with every batch, so run those streams on a slow cadence, weekly or monthly, and lean on the dedup views below. Run the append-cursor streams, which are incremental rather than full snapshots, as often as you like.

The post-edit-effort stream is a snapshot too, but a sparse one: it carries only edited segments, one row per editing episode, so its growth scales with your editing volume rather than your corpus size. A shop with a modest daily post-edit volume can run it daily for a trivial row cost; a heavy post-editing operation should run it on the same slow cadence as the full snapshot streams instead.

Querying the snapshot streams: the dedup views#

Each snapshot batch appends a full snapshot tagged with a batch id and timestamp. To query "latest state per row," ship a dedup view once.

Current-state streams: latest state per row

CREATE OR REPLACE VIEW <catalog>.<schema>.segments_latest AS
SELECT * EXCEPT (rn) FROM (
  SELECT *, ROW_NUMBER() OVER (
    PARTITION BY segment_id ORDER BY snapshot_at DESC) AS rn
  FROM <catalog>.<schema>.segments)
WHERE rn = 1;

The same shape works for the translation-memory and post-edit-effort streams, partitioned by their own row keys.

Quality findings: latest scoring per document and locale

The quality-findings stream is append-cursor, not a snapshot, but it is not insert-immutable: a re-score deletes a document and locale's prior findings and re-inserts fresh rows, so both generations persist side by side with no marker distinguishing them. The correct view ranks scoring events, then joins back to keep every row of the winning one:

CREATE OR REPLACE VIEW <catalog>.<schema>.lqa_findings_latest AS
WITH cards AS (
  SELECT document_id, target_locale, segment_id, scoring_id,
         MAX(created_at) AS scored_at
  FROM <catalog>.<schema>.lqa_findings
  WHERE scoring_id IS NOT NULL
  GROUP BY document_id, target_locale, segment_id, scoring_id),
ranked AS (
  SELECT *, ROW_NUMBER() OVER (
    PARTITION BY document_id, target_locale, segment_id
    ORDER BY scored_at DESC, scoring_id DESC) AS rn
  FROM cards)
SELECT f.* FROM <catalog>.<schema>.lqa_findings f
JOIN ranked r ON f.scoring_id = r.scoring_id
WHERE r.rn = 1;

A quality-now consumer, such as a scorecard or dashboard, wants this view. A trend or history consumer, asking whether quality improved after a change, must query the base table instead, since the view intentionally collapses exactly the history it needs. Rows written before this scoring-identity concept existed carry no scoring id and are excluded from the view by construction; a customer with older data will see fewer rows in the view than in the base table, which is expected, not a bug.

Delivery semantics and honest boundaries#

• Effectively exactly-once, with zero customer setup. Batch paths are deterministic, not random, so a retry after a network failure, a retry after a load whose response was never seen, and a crash between upload and load are all safe: the load step skips already-loaded files by identity.
• Loads are cast from the versioned schema, never inferred from the file. This avoids a real class of failure where an all-null column in one batch gets inferred as the wrong type and then fails to merge into a table's declared type. New columns in a later release still merge in additively; type inference drift cannot happen.
• Snapshot streams do not reuse a path, by design. Each snapshot batch writes to a new, uniquely named location, so a lost-response retry appends a second full snapshot; the dedup views above collapse it back to latest-per-row. That is intended, not a duplicate bug.
• A stream's cursor advances only on that stream's own confirmed load. If one stream's load fails while others succeed, the run is partial: the succeeded streams advanced, the failed one did not and re-exports next run. If every data-bearing stream fails, the whole run is reported failed, never dressed up as partial.
• Pick one export regime and stay in it. A local staging-only run and a real upload keep fully independent frontiers, and mixing an unrestricted export with a scoped export of the same rows causes each to keep its own high-water mark, so an overlapping row can land twice under two different regimes. It is lossless, just duplicative if you mix regimes; the two streams without a dedup view need de-duplication downstream if you do.
• Deletions do not propagate. The lakehouse is an append-only shadow copy; it cannot see a source row vanish. Erasure or deletion inside Localization OS does not propagate to your lakehouse. Your own retention or erasure obligation on the lakehouse side is yours to run. Each run stamps a flag marking that deletions are not tracked, and records per-stream counts for rows whose parents or themselves were already gone at export time.
• A stalled, long-running step is bounded. A non-terminal workflow-step row, such as a paused human review, holds the export frontier only for a limited, configurable window; an older one is skipped past and counted separately, so one paused review can never freeze the stream indefinitely.

Testing#

The full test suite for this exporter runs with zero live Databricks connectivity, against documented response shapes. A separate live verification runbook against a real workspace is opt-in and skipped by default; it proves the grant list above is complete against a real environment, while the offline suite proves the client's own logic.