S3-compatible file storage

Localization OS stores upload blobs, the source files behind every document, behind a storage seam. The default writes them to local disk. This optional adapter points that same seam at an S3-compatible object store instead, without any application code change. It is dormant unless you opt in: a default install imports nothing new and keeps using the disk store.

The at-rest encryption envelope used for uploads is byte-identical across the disk and S3 adapters, so a blob written encrypted on disk reads back correctly from S3. That is what makes the manual migration below a plain byte copy.

What you need#

• An S3-compatible bucket you control.
• Static credentials: an access-key ID and secret access key, optionally a temporary session token. This adapter is static-credentials-only today; instance-profile-style credential chains (IAM instance profiles, ECS task roles, EKS IRSA) are not supported yet.
• The S3 client library installed on the host, as an opt-in extra:

  pip install boto3

  It is not part of the default install; a default install never needs it. If you turn the switch on without it, the app fails to boot loudly and tells you to install it, never a silent fallback to disk.

Configuration#

Non-secret configuration is stored as Settings, set with the CLI's config set command or the Settings page. Credentials are host environment variables only: never Settings, never persisted to the database, never logged.

Settings

The bucket, endpoint, and the master switch are host-only settings: they decide where the install's data physically lives, so an admin key alone can never move them over the API. The region and prefix stay settable over the API, since a region cannot point at unrelated infrastructure and a prefix stays inside the already-fixed bucket.

A non-empty prefix gets a trailing slash appended automatically, so a prefix and a key never fuse together unexpectedly. An empty prefix stays empty. A handful of malformed prefix shapes, such as a leading slash or a parent-directory segment, are rejected loudly at boot rather than silently normalized.

The endpoint URL must be a plain scheme://host[:port] address; one that embeds credentials in the URL itself is rejected at boot. Provide credentials only through the host environment variables below.

Do not point two installs at the same bucket and prefix. Object keys are unique within one install's database, but not globally; two installs sharing a bucket and prefix can collide on overlapping ids and silently overwrite each other's blobs. Give each install its own bucket, or its own distinct prefix.

Credentials (host environment variables)

Endpoint examples

IAM policy: s3:ListBucket is required#

Grant the principal the minimum policy below. s3:ListBucket is not optional: without it, S3 returns a generic access-denied error instead of a clean not-found for a missing object, which hides an object's existence from a principal that cannot list. Localization OS treats that access-denied response as a loud infrastructure error, since a wrong-bucket or permission fault should never be quietly reported as "missing," so a policy missing ListBucket would make every read of a legitimately-absent blob fail loudly instead of behaving like a normal missing file. Grant ListBucket and a missing object correctly returns not-found.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::YOUR_BUCKET/*"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::YOUR_BUCKET"
    }
  ]
}

ListBucket is a bucket-level action; its resource is the bucket itself, not the wildcard path inside it.

Migration runbook (manual)#

Relocating existing blobs from disk to S3 is a manual runbook. There is no migration button and no background backfill yet.

1. Stop the app, or accept that in-flight uploads during the window land on disk.
2. Copy the uploads directory into the bucket at the normalized prefix destination. Copy raw bytes, names unchanged, using any S3-compatible sync tool. Because the encryption envelope is byte-identical across adapters, encrypted files copy over and read back correctly; there is no decrypt or re-encrypt step.

   aws s3 sync data/uploads/ s3://YOUR_BUCKET/loc-uploads/

3. Grant the IAM policy above, then set the configuration and credentials on the host and restart. Set the master switch on last, after the bucket, region, endpoint, and credentials are already in place, so you never leave the install in an enabled-but-misconfigured state:

   config set blobstore_s3_bucket YOUR_BUCKET
   config set blobstore_s3_prefix loc-uploads
   config set blobstore_s3_region us-east-1
   # (endpoint_url only for MinIO/R2/B2)
   export LOC_TMS_S3_ACCESS_KEY_ID=...
   export LOC_TMS_S3_SECRET_ACCESS_KEY=...
   config set blobstore_s3_enabled 1   # last: flip the switch once everything else is set

   If you do end up enabled-but-misconfigured, the app refuses to boot and blob-touching CLI commands fail loudly, but configuration and database commands stay usable, with a warning, so you can fix the configuration or turn the switch back off.
4. Verify a known document downloads and exports correctly, then, optionally, archive the local uploads directory.

Failure behavior#

The adapter is built to fail loudly rather than fall back silently:

• Enabled but misconfigured means the boot fails, loudly. If the master switch is on but the bucket is unset, credentials are missing, or the client library is not installed, the app does not start. A misconfigured S3 install never boots into a state where it silently writes to disk while you believe S3 is active.
• A transient network blip is a loud server error, not "your file is gone." A missing object returns the normal missing-blob path; a network or permission failure surfaces as a distinct server error, so a blip is never mistaken for a signal to delete and re-upload.
• No silent fallback to disk anywhere. Once the S3 adapter is active, the disk store is never a fallback for a failed S3 operation.

Orphaned-blob reconciliation

Delete is best-effort: on an operational S3 failure it logs a warning naming the object and bucket and returns without raising, rather than aborting a data-erasure sweep mid-loop. This means a blob may still exist in the bucket after an erasure or retention purge reported the corresponding row gone, an orphaned-blob risk inherent to best-effort delete over a remote store. After a purge or erasure against an S3 store, reconcile orphans on the bucket side yourself, for example by grepping the logs for the delete-failure warning to find the object key, or by running your own bucket lifecycle or reconciliation job.

Limitations#

• Static credentials only, today. Instance-profile-style credential chains, which resolve credentials with no static keys, are not supported; an enabled boot with no static keys fails loudly rather than silently trying to resolve one. Provide the access-key ID and secret access key on the host.
• Server-side encryption is your bucket's concern. It is an additive, optional second at-rest layer under Localization OS's own encryption envelope. Localization OS neither sets nor requires it; configure server-side encryption, bucket policies, lifecycle, and object lock on the bucket yourself if you want them.
• Credential scrubbing has a practical floor. The error-log scrubber protects genuinely secret-shaped values; very short placeholder secrets used only for local testing are not guaranteed to be caught the same way. Do not treat a short throwaway secret as redaction-protected. Real access keys and session tokens are well above that floor and are scrubbed.
• Bounded connection timeouts. The S3 client uses short, explicit connection and read timeouts instead of the client library's much longer defaults, so a black-holed or firewalled endpoint cannot stall startup or a worker thread for minutes. Set the endpoint and region correctly and a healthy store is unaffected.
• Single-part uploads, no content type. Blobs are written with a single upload call, no multipart, so an individual object larger than 5 GiB fails loudly rather than silently truncating, well above any realistic upload here. Objects carry no content type; they are opaque application blobs read back only by Localization OS, so this is intentional.

CLI parity#

The S3 adapter is registered for the CLI exactly as it is for the web server, so document upload, data-erasure, and retention-purge commands read and write the same S3 store as the web app, never the local disk. A remote-mode invocation against a running server is exempt: it is a thin HTTP client of that server's API, and the server owns its own storage.

When the install is enabled but misconfigured, for example the switch is on but the bucket is unset, the credentials are missing, or the client library is absent, a blob-touching CLI command fails loudly with a clean error instead of silently writing to disk, but configuration and database commands stay usable: they print a warning to standard error and run without S3, so you can fix the configuration or turn the switch off and retry. A machine-readable invocation of those commands still emits valid output on standard out, with the warning going to standard error instead.