Skip to main content

​
What “instance scope” means

An instance bundle is the whole Crewship in one file:
  • a page-level snapshot of the whole database (SQLite’s online backup API) — every workspace, user, credential row, setting and history;
  • every file store next to it: attachment files and workspace files, the memory trees and their version blobs, page projects, chats and skills, each file listed with its SHA-256 in the payload’s index;
  • every workspace’s crew containers (workspace, memory, home and tools, per the scope level);
  • optionally, the recovery kit: the vault keys the database’s sealed values need (see below).
It is restored offline with crewship recover into an empty data directory. Crew-scope and workspace-scope backups are covered in the Backup & Restore guide; this page is about what the instance scope adds.

​
Access control

  • Instance administrators only. Every instance backup route sits behind the instance gate: the CREWSHIP_OWNER_EMAIL break-glass owner, or anyone an instance admin named (users.instance_role = 'ADMIN'). An install that named nobody named the owners of its oldest workspace once, the first time anyone’s status was asked, recorded it in the instance audit (instance.admin_bootstrapped) and never does it again — removing every named admin later does not hand the instance to that workspace’s owners. An OWNER or ADMIN of a single workspace is refused with 403. CLI tokens need the instance:admin scope.
  • Per-workspace backups follow the same rule. The /api/v1/admin/backups* routes act on the workspace the request names and admit its OWNER or ADMIN, or an instance administrator who is not a member of it. Nobody else reaches another workspace’s bundles.
  • One run at a time. A second run while one is going answers 409.
  • Always encrypted. An unencrypted instance bundle is refused. Prefer age recipients (age1… public keys whose private halves live offline). A passphrase is accepted, but with the recovery kit on it is then the only thing between a copied bundle and every secret in the instance: use a long, generated one.

​
The recovery kit

Sealed values (credentials, webhook secrets, integration keys) are AES-256-GCM envelopes stamped with the key version that sealed them (v1:, v2: …). The keys live in the environment (ENCRYPTION_KEY, ENCRYPTION_KEY_V2 …), never in the database, so a database backup alone restores credentials nobody can read. With the recovery kit on (crewship admin instance backups recovery-kit on, off by default, audited), every key version an envelope in the database references rides inside the age-encrypted payload of each instance bundle, as recovery-kit/keys.json, byte for byte as the source had it (the journal hash chain key is derived from the ENCRYPTION_KEY string itself). Workspace and crew bundles never carry it. crewship recover writes the keys to <data-dir>/recovered-keys.env (mode 0600). Whoever holds a private backup key can read every secret in the instance. Store the age private key on a different device from the one that keeps the live ENCRYPTION_KEY, and delete recovered-keys.env once its lines are in your secret store. GET /api/v1/admin/instance/backups/vault-keys shows, per version, how many sealed values use it and whether this server has its key — counts only, never key material. A referenced version the server cannot resolve is recorded in the bundle as vault_key_missing rather than silently left out.

​
Crypto chain

  • Bundle alone: the payload is age-encrypted.
  • Bundle + age key, no kit: the database opens, sealed values do not — the attacker still needs the source’s ENCRYPTION_KEY.
  • Bundle + age key, kit on: full compromise of every secret in the instance. That is the price of a restore that works without the source host.

​
The quiet window

Release 1.0 requires one writing server per persistent local database, enforced by a process-lifetime OS writer lease before database startup. A second writer is refused; network/shared storage is not supported. Stop all old daemons before upgrading, because older binaries do not participate in this protocol. The quiet window checks the same owner before copying and at the end of the copy. Lost ownership aborts the backup without publishing its staged bundle; health/readiness report writer_ownership: lost and the server stops. See the single-writer decision. A consistent copy needs the database and the files to agree. The server first drains running work (agent runs, started work items, routine runs; up to two hours, then the run is skipped), then holds every write to protected data for the copy only: mutating HTTP requests answer 503 with Retry-After (backup, instance-hold and sign-in routes stay open), inbound webhooks answer 503, schedulers and queue consumers do not fire, and retention and GC sweepers wait. A hard cap (20 minutes) releases the hold and abandons the copy. The hold lives in memory only: a crash mid-copy comes back with nothing held. Packing and encryption run after the hold is released; the run records hold_ms.

​
Restore: sessions, keys and holds

crewship recover runs offline and:
  • drops every sign-in session and pairing, clears backup locks, the scheduler lease and provider-login leases;
  • writes a new NEXTAUTH_SECRET and CREWSHIP_ADMIN_TOKEN_HMAC_KEY into <data-dir>/secrets.env, so no token issued by the source works on the restored server (the rotation is enforced, not advisory);
  • clears instance_config.hostname, which the new host fills in at first boot;
  • holds routines and schedules, inbound webhooks and queued work (including notification retries) in instance_holds, so nothing fires, answers or retries until an instance admin resumes it (crewship admin instance holds resume <key>, audited).
Host-level container settings in the restored crews’ configuration — Docker socket mounts, privileged mode, host path binds — are reported by the restore checks as not carried over for safety, to be switched back on by hand after review.

​
Rotation policy

  • age recipient: rotate whenever a holder of the private key leaves. Bundles sealed to the retired key should be re-sealed or retired.
  • ENCRYPTION_KEY: rotate at least yearly, or immediately on suspected host compromise, with versioned keys (CREWSHIP_ENCRYPTION_KEY_VERSION, ENCRYPTION_KEY_V2 …) and crewship admin reencrypt. With the recovery kit on, bundles taken before the rotation still carry the old versions.
  • NEXTAUTH_SECRET: rotated by every crewship recover; rotate by hand if an operator session token leaks.

​
Incident response

If an instance bundle is known or suspected to have leaked:
  1. Rotate the age recipient holding the matching private key.
  2. If the bundle carried the recovery kit, treat every secret in the instance as exposed: rotate ENCRYPTION_KEY (a new version plus crewship admin reencrypt) and the upstream credentials themselves.
  3. Force-logout every session by rotating NEXTAUTH_SECRET.
  4. Review crewship admin instance audit for instance.backup_* entries in the incident window.

​
Known limitations

  • A live compromise before the backup is out of scope: root on the source is inside the trust boundary.
  • MANIFEST.json is plaintext so inspect works without a key. It carries no secrets, but workspace names and slugs, hostnames, file counts and key version numbers are visible.
  • Recovery goes into an empty data directory only (--force replaces an existing one). Merging an instance bundle into a running server is not supported.
  • Crew container files are staged under <data-dir>/restore-staging/crews/; they are not landed in containers automatically.