Skip to main content

​
Backup & Restore

Format v4 captures persistent quota-service disks through the configured host quota helper. Crew, workspace and whole-instance backups refuse live quota declarations when that transport is unavailable; they never publish a database-only archive that promises missing service data.

​
Overview

Crewship’s backup system produces portable .tar.zst bundles that capture a workspace (or a single crew) in one file. Bundles are AGE-encrypted by default, carry a versioned manifest, and can be restored on any Crewship instance that speaks the same format version (N-2 compatibility guarantee). The whole subsystem is admin-only by design: every backup subcommand requires the OWNER or ADMIN role on the workspace, and the runner refuses MEMBER / VIEWER calls at both the CLI parsing layer and the server-side handler — defence in depth, not just a UI veneer. The architectural choice that shapes everything else is “one bundle, one file”. Backups don’t depend on an external object store, don’t require a sidecar, and don’t shard across files. A .tar.zst is a single artefact an operator can scp to a backup host, hand to a customer for legal hold, or check into a private bucket — without the rest of Crewship being available. Inside the tarball, the manifest is plaintext JSON (so crewship backup inspect can read it without the AGE recipient key) and the payload is the encrypted SQLite snapshot plus any referenced workspace files. A reader at format N accepts bundles at N, N-1 and N-2; a newer bundle requires a reader that supports its format. older bundles run their migration’s restoreBackfill hook so columns added since the snapshot are populated sanely instead of left at SQL defaults. Restores are intentionally cautious. --dry-run walks the entire restore plan — schema diff, row counts, blob deltas — and prints what would happen without writing a single byte to the destination database. Advisory locking (backup_locks table, per-workspace) prevents two concurrent restore runs from corrupting each other, and the lock file records the host + PID so crewship backup status can tell you who’s holding it. If a host crashes mid-restore, crewship backup unlock is the manual recovery — admin-only with a confirmation prompt, because clearing a real lock from another live process is how you trash a workspace. Quota-enforced services with persistent volumes are captured through the configured host quota helper. When that transport is unavailable, bundle creation fails at every backup level with an explicit error rather than omitting those disks. Legacy service storage retains its existing backup behavior.

​
When to use it

Backups are cheap to make and expensive to wish you’d made. The five canonical reasons to run crewship backup create:
  • Before any destructive admin write. Before crewship admin reset-password, before a database migration on a binary upgrade, before a large schema-changing PR lands in prod — capture the current state first. The bundle is the one-command rollback if anything goes wrong.
  • Disaster recovery / hot spare. Schedule a nightly crewship backup create --scope=workspace --passphrase-file … cron and ship the bundle to a separate host. If the primary disk dies, restoring onto a fresh binary is one crewship backup restore away — no replication agent, no streaming WAL, no extra moving parts.
  • Workspace migration to another host. Moving a workspace from a dev VM to a prod host (or between two prod hosts) is exactly what bundles are for. Create on the source, scp the file, crewship backup restore --as-workspace <new-slug> on the destination, start the crews, then crewship backup restore … --files-only to land their filesystem state. The --as-workspace rename avoids the “two acme workspaces colliding” hazard; the third step is not optional — without it the rows migrate and every crew’s workspace files and memory tree stay behind.
  • Legal hold or compliance archive. Customer leaves; you need to keep their workspace state on cold storage for N years. One AGE-encrypted .tar.zst is a forever-readable artefact — no live database, no service required. inspect later proves the bundle’s contents without decrypting.
  • Forensic snapshot before incident response. Suspected compromise of an admin account, or a “what was the state when X happened” investigation. backup create + immediate offsite copy preserves the audit trail before anyone (you, the attacker, the well-meaning oncall) starts changing things.
Skip backups for ephemeral development workspaces (the dev VM rebuilds them from seed scripts anyway) and single-agent throwaways with no user-visible state worth preserving — the bundle metadata cost is real (~few MB minimum) and not every workspace earns it.

​
Key concepts

​
Usage

The whole backup surface is the crewship backup command group — twelve subcommands that cover the create → verify → restore → rotate lifecycle (plus metrics, download, and self-test). The core nine are tabulated below with full flag reference and copy-pasteable examples; metrics is covered under Metrics, download under Known caveats. This table is the entry point. The minimum end-to-end loop is four commands: create to produce the bundle, verify to prove it’s not corrupt, restore --dry-run to prove the destination will accept it, then restore for real. Every other subcommand exists for retention (rotate), introspection (list, inspect, status), or recovery (unlock).
create and restore accept --use-keyring to cache and reuse the workspace passphrase via ~/.crewship/backup-keyring.enc. See Passphrase keyring below.

​
Bundle layout

  • Scope: workspace (workspace row + all its crews) or crew (single crew + its agents).
  • Encryption: AGE passphrase (default) or AGE X25519 recipient — every new bundle is encrypted, because agent home folders inside a bundle can hold credentials the vault never sealed (~/.aws, ~/.ssh, tool logins).
  • Default location: $CREWSHIP_DATA_DIR/backups/ on the server (mode 0700), or ~/.crewship/backups/ when CREWSHIP_DATA_DIR is unset — the same root every other piece of per-instance state resolves against, so isolated instances on one host never share a bundle directory. --output/output_dir overrides it explicitly.
  • Naming: crewship-<scope>-<slug>-<iso-ts>.tar.zst. Collisions append -<hash8>.
Inside payload.age, bundles carry three distinct notions of “memory” — don’t confuse them:
  • Per-crew agent and crew-shared memory (crew/<slug> section) — the real memory tree: /crew/shared/.memory and /crew/agents/<agent-slug>/.memory inside the crew container. This is what an agent accumulates and what is hardest to regenerate. Collected by the docker phase, at every scope level including quick.
  • Per-crew declared output (memory/<slug> section) — the container’s /output mount. The section name is historical; it has never held agent memory.
  • Workspace-tier memory-version blobs (memory-blobs/<sha[:2]>/<sha> section) — the content-addressed audit trail behind memory_versions rows (consolidation output, HITL-approved proposals, pin snapshots). Collected once per workspace scope from {MemoryRoot}/versions, keyed by the sha256 the DB dump’s memory_versions rows reference. Restore rewrites payload_ref to the target’s blob root — see the Memory-version blobs row under Key concepts above.
A workspace bundle whose pages carry custom applications has one more section, page-projects/, sealed in the same payload:
  • page-projects/sources/<sha256>.yaml — every source project a draft, revision, build or publication row still points at, as the same YAML crewship page project pack prints.
  • page-projects/artifacts/<sha256>.json — every compiled artifact a build or publication row points at.
  • page-projects/git/<page-key>/<kind>/<id> and …/shallow — the Git objects behind the retained checkpoints, truncated at the retained checkpoint boundaries so the bundle carries what crewship page project history can restore and nothing older. <page-key> is the sha256 of the page id; restore maps it onto the target’s page ids, so a renamed restore lands it too.
This section is written under a shared lease on the page project store, held from before the DB dump, so no save or build can move a pointer between the dump and the files it references; a source or artifact the rows name but the disk no longer holds fails the backup rather than producing a bundle that would fail to restore. The section only exists when the page_project_* rows reference something, and it needs CREWSHIP_PAGE_PROJECTS_PATH on the instance taking the backup — a bundle whose rows reference sources with no path configured is refused at create time. On restore, the whole section is verified in a staging directory — every checksum, every checkpoint against the definition the row recorded — before the database transaction, and applied under an exclusive lease before the commit; a dry run verifies it too. A bundle whose page_project_* rows have no matching section is refused as incomplete, and one whose section references nothing the rows name is refused as well, so a database-only restore can never leave a live publication pointing at code the instance does not hold.
Bundles at format_version 1 or 2 contain no agent or crew-shared memory, and their manifest says memory_included: true anyway. Up to and including v2, the collector read /output and filed it under the memory/ section, and the flag was set from “this crew had a container” rather than from anything that was actually captured. Everything else in such a bundle — workspace files, named volumes, DB rows, memory-version blobs — is real and restores normally, and the restore prints a warning naming the crews whose memory is not in there. Memory from before your first v3 backup cannot be recovered from an older bundle. Take a fresh backup after upgrading.
From format_version 3 onward every per-crew inclusion flag in the manifest is derived from what the collector observed itself writing, not from what it expected to write. A section that captured nothing reports false, which is also what stops the restore preflight from demanding a provisioned container for a crew that has no data.

​
Attached files, and what a bundle records as missing

Files attached to issues, comments and chats live on the host under the storage root, one content-addressed file per upload, outside the database and outside every crew container. A workspace bundle carries each one its attachments rows reference, in an attachment-blobs/ section inside the encrypted payload. Bundles written before this section existed carried the rows and none of the files: restoring one lands attachments whose downloads fail. On restore each file is written back under the workspace the restored row belongs to — the new workspace on a --as-workspace fork — and its SHA-256 is checked before it is put in place. A file already at that path is never overwritten: identical bytes count as already present, different bytes are left alone and reported as a conflict. Nothing a bundle lacks is silent. When a file the rows reference is not on disk at backup time (or attachment storage is not configured), the backup still completes, and the manifest records it: contents.attachments_included, contents.attachments_missing, and contents.incomplete — one item per kind of gap (attachment_missing, memory_blob_missing, container_missing for a provisioned crew whose container was gone, environment_failed for a crew whose complete environment could not be captured), each with a count and a sentence. The restore response repeats them with what the restore itself found (attachments_restored, attachments_already_present, attachments_missing, attachments_conflicts, incomplete, and environments — each crew’s environment restored, rebuilt or skipped), and a restore with any of them is recorded as partial, not ok. The section is additive, so it did not change the format version: an older server restores such a bundle exactly as before, rows without files.

​
Which rows a bundle carries

A workspace bundle is not “the whole database minus other workspaces” — it is one SELECT … WHERE … per table, and the WHERE is derived, not hand-written. DiscoverScopedTables walks the schema’s foreign keys backwards from workspaces and records, for each table, the chain of hops that ties one of its rows to a workspace. crews is one hop (workspace_id); agents is two (crew_id → crews.workspace_id); agent_skills is three. Most tables have more than one such chain, and which one is chosen decides which rows exist as far as the bundle is concerned. Two rules govern the choice:
  • A NOT NULL chain beats a shorter nullable one. A filter on a nullable column silently omits every row where that column is NULL — and for a column like “the run that produced this” or “the agent who claimed this task”, NULL is the ordinary state, not the exception. The walk therefore makes a first pass that follows only NOT NULL foreign keys, so a three-hop total path is preferred over a one-hop lossy one. A table that has no NOT NULL route out at all needs an explicit filter in workspaceFilterSQL, and TestScopedFilters_NeverTraverseANullableFK fails the build until it gets one.
  • Ties are broken deterministically. Two parents the same distance from workspaces used to race through Go map iteration order, so the same schema could produce different filters on consecutive runs. The walk is now level-synchronous over a sorted adjacency, so a bundle’s contents depend on the schema and nothing else.
The practical consequence for an operator: a table added by a migration is picked up automatically, but only once it declares an intent in BackupTableIntent. Until then, restore-time drift detection refuses rather than guessing — see migrations. memory_included counts files inside a .memory directory, not tar entries. The distinction matters because the container runtime creates crews/<id>/shared and crews/<id>/agents when a crew container is created, so counting entries would report memory for every provisioned crew whether or not an agent had ever written a note. crew_files_included is the separate flag for “the /crew section carries something” — init.sh and other crew-level content — and is what decides whether the section is restored at all.

​
Create a workspace backup

The CLI prompts twice for a passphrase (to guard against typos) and confirms success with a row summarising scope, size, format version, and the SHA-256 of the sealed payload.

​
Non-interactive / CI

Supply the passphrase from a file:
Or pipe a single line on stdin (falls back automatically when stdin is not a TTY and --passphrase-file is not set).

​
Asymmetric encryption

If the restoring party holds an AGE X25519 keypair, pass their public key instead of a shared secret:
--recipient and --passphrase-file are mutually exclusive. --no-encrypt is refused: every new bundle is encrypted.

​
Back up a single crew

--crew accepts either a slug or a crew ID. Crew-scope bundles restore independently of their parent workspace.

​
What happens to the crew container

A backup freezes each crew container only as far as the container’s own state allows, and puts it back exactly the way it found it (#2612): Restore is the one flow with the opposite requirement: it writes through tar inside the container, so the target crew must be running — see The target crew must be running.

​
List, inspect, verify

inspect only reads the plaintext MANIFEST — it never touches the sealed payload, so no passphrase is needed. verify recomputes the SHA-256 of the sealed bytes against the manifest and fails if the bundle was truncated or tampered with — that check never decrypts an encrypted bundle.
What verify checks (#2009). It always checks integrity: “are these the bytes we wrote?” For an unencrypted bundle it also checks completeness — every table’s actual row count in the payload against manifest.contents.table_row_counts, recorded at create time — and now reports INVALID on a divergence, not VALID:
An encrypted bundle’s completeness cannot be checked by verify — decryption needs the passphrase, and verify deliberately never asks for one, which is what lets it run unattended against a whole directory of nightly bundles. It still reports checksum-only VALID, but says so:
restore --dry-run already decrypts and parses the payload for other reasons, so it makes the identical comparison and reports it as payload_row_count_mismatches — see Restore below. A bundle written before #2009 carries no recorded counts at all; verify reports the same “NOT verified” shape for it, never a bare VALID that implies the check happened.The recorded count is len(dump.Tables[table]), read from the same in-memory dump that is written into the payload, so on its own it only proves “the bundle is intact”. A create-time scoping bug that silently drops rows (the shape of #1973) produces a dump that agrees with itself.“The bundle is everything” is checked separately. A workspace backup derives, inside the dump’s own read snapshot, which rows the schema ties to the workspace: rows whose own workspace_id names it, and rows whose foreign key points at a row the dump exported from a workspace-owned parent. Global tables (users, skills) do not pass ownership on. It then compares those rows with what the scope filters exported. The result is recorded as manifest.contents.scope_reconciliation (checked, tables, shortfalls). Because the manifest is readable without the passphrase, verify fails any bundle that records a shortfall, encrypted or not:
A shortfall means a scope filter in the backup code left rows behind. The bundle restores what it carries, but not everything the workspace held, so report it as a bug. Bundles written before this check, and crew- or instance-scope bundles, carry no scope_reconciliation. A failure to reconcile never fails the backup itself: it is recorded as checked: false with a skip_reason.
An instance bundle belongs to no workspace, so the per-workspace routes answer 404 for it; inspect, verify and download then read it through the instance routes (GET /api/v1/admin/instance/backups/bundles/inspect, …/verify, …/download), which serve any catalogued bundle to an instance administrator. Backup history’s Download uses the same route for an instance bundle, and each download is recorded in the instance audit log.

​
Restore

The server rejects the restore if a workspace (or crew) with the same slug already exists. Override with --as-workspace <new-slug> or --as-crew <new-slug> to land the payload under a fresh identity.

​
Restoring under a new identity takes three steps

A rewritten restore forks the bundle’s rows under a new workspace id and slug. It cannot land container state in the same call, because the crews it just created have no containers yet. So the flow is:
--files-only restores the per-crew filesystem sections (workspace, memory tree, named volumes, /var/lib) and changes no database rows. It is authorised by provenance, not by the flag: step 1 records which bundle the new workspace was forked from — matched on the bundle’s payload digest — and step 3 is permitted only for a workspace that record covers. A bundle that was never restored into your current workspace is refused exactly as it was before. That provenance authorises only --files-only. Running the plain restore against a forked workspace is refused, and deliberately so: without a rewrite flag it takes the ordinary docker phase, whose crew identities come from the manifest — on a same-instance restore those are the source crews, and it would overwrite their live workspace and agent memory with the older backup while INSERT OR IGNORE kept the row counts looking untroubled.
Step 3 is what earlier versions described as “re-run restore without the rewrite flag”. That never worked and could not: the forked workspace’s id and slug match neither of the bundle’s, so the tenant guard rejected it every time. --files-only is the step that replaces it. It also addresses the crews the fork created rather than the crews the bundle names — the manifest still carries the source crew’s slug, so on a same-instance restore the older advice would have written a sibling’s backup over the source crew’s live data.
Restoring under the ORIGINAL identity (same workspace, or a genuinely empty instance) is unaffected: the docker phase runs in the same call and there is no step 3. A fork regenerates identity, not merely primary keys: alongside every row’s id it also mints a fresh value for the columns that have to be unique across the whole instance rather than within a workspace — a mission’s trace_id is the one that exists today. Without that the fork’s row would collide with the source row still holding the value and be dropped in silence, taking its children down with it (#2260). Columns that are unique only within a workspace are left alone on purpose: an issue identifier like ENG-14 is scoped by the workspace id the fork already remapped, so the fork keeps the identifiers your team knows.

​
A fork does not inherit live capabilities

Workspace invitations, port exposures, pipeline webhooks, page public links and page webhooks are authorised by a capability token — a secret whose knowledge is the whole of the check. A fork re-keys every one of them (#2274): the token and its digest are minted fresh, together, through the same digest primitive the auth layer verifies with, so no secret that worked against the source works against the fork. Anyone who ever learned a source URL — including from a leaked database file — holds nothing on the copy. What lands, and what you have to do: The restore says so, on the committed path and on a dry run alike: a capability_tokens_reminted note per table in the CLI output, the API response and the backup.restore audit row. A webhook that arrives revoked is a fact to act on, not a footnote. Two related keys stopped being instance-global at the same time: inbox_items’ dedupe key and message_feedback’s feedback key are now scoped per workspace, so a fork keeps its inbox and its message feedback instead of silently losing every row to the source’s (#2274). Nothing about single-workspace behaviour changed — one user still cannot file the same signal twice on the same message, and a producer still gets one open item per source.

​
Forked restores re-sign the journal chain

The crew journal is tamper-evident: every entry carries a keyed HMAC over its own content and the hash of the entry before it, so an after-the-fact edit or a mid-chain delete is detectable (see Audit & tamper-evidence). Identity columns are inside that HMAC — the entry’s own id and its workspace id among them — and --as-workspace / --as-crew regenerate exactly those. A fork therefore cannot keep the bundle’s signatures: they attest to ids that no longer exist. So the restore recomputes them, and the forked workspace gets a new genesis under this installation’s chain key. What that means in practice:
  • The fork’s journal verifies clean. crewship journal verify -w <fork> (and the admin integrity endpoint) report ok immediately after the restore, instead of calling every restored row tampered.
  • The fork’s chain attests to this instance, from this restore onward. It carries no cryptographic link back to the source workspace. If you need to prove the source’s history, verify it on the source (or on a --replace / same-identity restore, which remaps nothing and keeps the original signatures intact).
  • A backup.chain_resigned journal entry is written at the tail of the new chain, recording how many entries and compaction checkpoints were re-signed, the source workspace id, and the bundle digest. It is inside the chain it describes, so it cannot be removed without breaking verification. Do not treat its absence in a forked workspace as normal.
  • Signed compaction checkpoints follow the fork and are re-signed under the new workspace id, so a journal that was legitimately compacted before the backup does not read as a malicious mid-chain delete afterwards. They are no longer duplicated into the source workspace’s checkpoint set.
A restore that cannot re-sign the chain fails rather than landing a fork whose journal would report as tampered. The failure happens before any row is written, so nothing is half-applied. Rows that predate the hash chain (schema v152) were never signed; a fork carries them across exactly as they are, no better and no worse than the source. crewship backup restore reports the counts:
The same two counts come back on the API as journal_entries_resigned / journal_checkpoints_resigned, and land on the backup.restore audit row.

​
The target crew must be running

Every section is written through tar inside the container now (see Bundle layout), so a restore needs the crew container running, not merely created. A stopped crew is refused in the preflight, before anything is written, and the message names the fix:
crew start is the command for this: it starts the container, and provisions its image first if the crew has none — so it covers both the stopped crew and the never-provisioned one. It is not crew provision. Provision builds an image and starts nothing; on a cache hit it reports provisioned in seconds while the container stays stopped. This message used to name it, which sent operators round the loop once more before they got here. The preflight also refuses when a path the bundle writes to cannot be written by the agent — a directory or file left owned by root or by the memory sidecar. That is a refusal with nothing written, which is the point: the same restore used to fail partway through, after earlier sections had already landed. Hand the named path to the agent (chown -R 1001:1001 <path>) and re-run. Paths the bundle never touches are ignored, so unrelated root-owned files under /home/agent do not block anything.
One thing is reported without failing the restore: if the crew memory tree’s group and setgid bits cannot be fully re-applied afterwards — a .memory directory owned by the memory sidecar, say — the restore succeeds and warns, naming the directory. The data is on disk; only the sharing contract is degraded, and failing at that point would roll back the database while leaving the container’s files in place.

​
Dry run

A dry-run decrypts the bundle, validates the manifest, replays the DB transaction, and then rolls back. The only side effect is a single backup.restore.dry_run row in the audit log — handy for proving a bundle is restorable before the real cutover. The distinct audit action lets auditors tell “verified” apart from “actually restored”.

​
Credential security levels are re-checked on the way in

A restore writes the bundle’s rows as it finds them — with one exception. credentials.security_level is not data, it is the input to a Keeper gate, so every value is checked against the tier table (L1–L4) as the bundle is applied. A level the table does not define — 0, a negative number, a tier from a future release, a corrupt or tampered value — is written at the strictest tier instead:
Three things about that behaviour are deliberate:
  • The credential is kept, not dropped. Recovering an instance that is quietly missing a secret is a worse outcome than recovering one credential at too strict a tier.
  • The clamp goes up, never down. The strictest tier is what Keeper already resolves an unrecognised level to at every decision point, so no gate changes behaviour; what changes is that the stored value now agrees with the gate.
  • You are told either way. The count and the affected credentials come back in the API response, and a credential.security_level.clamped entry lands in the journal with the value the bundle carried — so the question “why is this credential L4?” has an answer months later.
--dry-run reports the same thing (as would be clamped) without writing, which is the cheapest way to check a bundle before a cutover.

​
Columns your schema does not have are reported, not just dropped

A restore applies only the columns that exist on this instance. That is what lets a bundle from a newer Crewship land on an older one — the newer columns are skipped and the rest of the row goes in. When it happens, the restore now says so:
Why it is worth a warning rather than a shrug: dropping a column is harmless when the row still makes sense without it, and it is silent data loss when it does not. If the dropped column was part of the row’s primary key or a NOT NULL, the insert violates a constraint, INSERT OR IGNORE discards the whole row, and nothing errors — the bundle verifies, the restore reports success, and the rows are simply not there. rows_inserted cannot show you the difference, because a row that never landed is not counted anywhere. The case that produces it is a bundle taken before a non-additive migration — one that re-keys or renames a table rather than adding to it. Restoring across one is not something the restore path can repair on its own in general: it does not know what the old column meant. What it can do is refuse to be quiet about it, so you check the tables it names instead of finding out months later. issue_counters (below) is the one table this restore does know how to repair, because #2034 taught it that table’s specific rename. --dry-run reports the same skew (as would be dropped) before anything is written.

​
A pre-#1797 issue_counters row is migrated, not dropped

issue_counters used to be keyed on crew_id; migration 20260820125000_issue_counters_prefix_scope (#1797, documented in the migrations guide) re-keyed it to (workspace_id, prefix). Without special handling that is exactly the shape of loss the previous section describes: crew_id is a column the new table does not have, the row degenerates into a constraint violation on the two new NOT NULL key columns, and INSERT OR IGNORE swallows it — every counter gone, silently (#2034). Restoring a bundle taken before that migration resolves each row’s crew — from the bundle’s own crews table, or from the target if the crew already exists there — to that crew’s workspace and effective prefix (issue_prefix, or the first three letters of the slug, upper-cased — the same rule the migration’s backfill and the runtime allocator both use), and writes the counter under its current key instead:
Two crews that share an effective prefix collapse onto one migrated row, keeping the higher next_number — writing back the lower one would let the allocator hand out an identifier that already exists, which is worse than the counter being absent (an absent counter is reseeded above whatever identifiers restored alongside it). A row whose crew cannot be resolved at all — the crew, like its issues, is also gone, and is not in this bundle or on the target — is not guessed at. It falls through to the ordinary column-drop report above, issue_counters.crew_id, so you still hear about it; the counter itself is the one thing that scenario cannot recover, because it was the crew’s only remaining record of how far it had counted. issue_counters_migrated in the API response (and --dry-run, as would be migrated) carries the count. Because that collapse happens before insert, rows_inserted_shortfalls (below) discounts issue_counters by the number of rows the collapse actually removed — the manifest’s pre-collapse count minus that discount is what a healthy restore is compared against, the same discount pattern rows_inserted_shortfalls already applies to users. A genuine issue_counters shortfall — a row whose crew could not be resolved, or an ordinary PK collision — still shows up.

​
Restore checks the payload against the manifest, and the target against both (#2009)

verify can only check completeness for an unencrypted bundle — it never decrypts. Restore already has the decrypted payload for other reasons, so it makes the SAME comparison verify does, regardless of encryption, and adds a second one for what actually landed:
That first warning — payload row counts — is the same divergence verify reports for an unencrypted bundle: the decrypted payload disagrees with what manifest.contents.table_row_counts claims. The second, separate warning — rows inserted — is about the TARGET, not the bundle: a table can come up short on the way in even when the payload itself is exactly what the manifest says, because a schema-skew column drop (previous section) or a primary-key collision INSERT OR IGNORE swallowed kept some of its rows from landing:
--dry-run reports the payload check (nothing was inserted, so there is nothing to compare for the second one). Neither is a hard refusal — a divergence here is exactly the kind of thing a disaster-recovery operator needs to see and decide about, not something the restore should silently work around or silently block on.

​
Delete & rotate

rotate applies retention per workspace — it never touches another workspace’s bundles, and crew bundles are counted separately from workspace bundles (and per crew), so one never pushes the other out. Either flag must be positive:
  • --keep-last N is a floor: the N newest bundles are always kept, however old they are. Beyond them, with no --keep-days, everything goes.
  • --keep-days D applies only beyond that floor: a bundle past the N newest is kept while it is younger than D days. With --keep-days alone the floor is one bundle — no rotation deletes the last copy.
  • A bundle pinned with crewship admin instance backups pin <path> is never deleted by rotation.
Before 2026-09-30 a bundle was dropped when it was beyond N or older than D days, so a workspace whose backups had stopped for longer than D days lost every copy at the next rotation. delete requires interactive confirmation, or --force in scripts / CI. The same rule applies to backup unlock.

​
Lock semantics

Each workspace holds at most one advisory backup lock at a time (table backup_locks, per-workspace PK). The lock:
  • Is taken before any DB dump or docker pause and released by a deferred Release() on the happy path.
  • Has a 1-hour TTL (DefaultLockTTL); a crashed backup self-heals after the window.
  • Blocks concurrent backup create calls — the second caller gets HTTP 409 Conflict with a “another backup is already in progress” message.
  • Blocks new agent runs via the shared refuseIfBackupInProgress guard wired into the assignments, peer-query, and webhook handlers. This closes the TOCTOU window between ensureAgentsIdle and docker pause.
Inspect or release the lock:
backup unlock is an emergency escape hatch. Only use it when you can confirm no backup is actually running (e.g. the previous CLI session crashed and the 1 h TTL has not yet fired). Forcibly releasing a live backup’s lock will let a second backup start alongside it, and the two will race on the docker pause/unpause sequence.

​
Examples

​
Nightly hot-spare backup with 14-day retention

A workspace on prod-server.example.com should produce a bundle every night, ship it to a separate backup host, and keep 14 days on disk locally as a fast-rollback safety net.
Paired with a crewship-backup.timer that fires at 02:37 daily (off-the-hour to avoid clustering with the rest of the fleet). The --use-keyring flag means the passphrase is read once and cached — the timer doesn’t have to redeliver it. The rotate step runs after the rsync so local-disk pressure is bounded even on long runs without a remote-side sweep.

​
Workspace migration to a new host

The acme workspace lives on crewship-old. You’re moving it to crewship-new to retire the old host. The destination already has its own workspaces, so a same-slug restore would conflict.
Once the new instance has parity (sanity-check via the UI), users get the new URL, and acme on the old host gets archived to cold storage before its workspace row is deleted.

​
DR drill with --dry-run

Every quarter the team proves the disaster-recovery bundle is actually restorable, without disturbing production state. Run the drill on a throwaway VM:
A backup.restore.dry_run audit entry shows up in the workspace journal — auditors looking at “did anyone restore prod?” can tell drill runs apart from real restores by the action name.

​
API reference

The backup surface is CLI-first — every operation is reachable via crewship backup … and every flag the CLI accepts maps to an HTTP body field. The full HTTP schema lives at /api-reference/backup; a quick orientation: Every route acts on the one workspace the request names (workspace_id in the query, or X-Workspace-ID; an id or a slug) and admits an OWNER or ADMIN of that workspace, or an instance administrator, who need not be a member of it. So an instance admin downloads, verifies and restores any workspace’s backups — crewship backup list --workspace lab, or Admin › Backups with that workspace ticked — without joining it; a workspace ADMIN still reaches only their own. A bundle of another workspace answers 404 through the wrong workspace. metrics needs no workspace and admits only an instance admin. All routes are mounted in internal/api/router_admin.go. The CLI talks to these directly when run against a remote host (--server flag) and falls back to in-process Go calls when run on the same machine as crewshipd — bypassing HTTP entirely for the host-shell use case. The two paths share the same handler functions, so flags work identically. Webhook payloads for backup.created / backup.restored / backup.failed events are documented separately under Webhooks; metric emissions are documented under Metrics. (Dry-run restores deliberately fire no webhook — they are not real restores.)

​
Streaming & memory bounds (large backups)

Restore and verify stream the sealed payload to a temp directory rather than buffering per-crew sections in a map[slug][]byte. Peak heap stays bounded by the zstd decoder window regardless of bundle size, so multi-GB restores run cleanly on small hosts. The extraction scratch directory is os.TempDir()/crewship-backup-… and is removed on Close(); a killed process leaves it behind for the next os.TempDir cleanup.

​
Passphrase keyring

The --use-keyring flag on create and restore caches the workspace passphrase in ~/.crewship/backup-keyring.enc so scripts and repeat rehearsals don’t re-prompt. The file is an AES-256-GCM-encrypted JSON map keyed by workspace ID, using the same v1:<base64> envelope as the credstore — without the host’s ENCRYPTION_KEY the contents are unreadable even if the file leaks.
Semantics worth knowing:
  • No silent fallback on failure. Opening the keyring or reading an entry reports the real error and aborts; only ErrKeyringEntryNotFound (first use on this workspace) falls through to a prompt.
  • Write failures are non-fatal on create. The bundle is already written when the keyring save runs; the CLI logs a warning and continues.
  • Keyring is local to the operator’s host. --use-keyring always writes to ~/.crewship/ on the invoking machine — even if a future remote bundle backend (S3 / GCS) is configured, the passphrase never travels with the bundle.
  • Single-process mutex, not file-locked. Two concurrent CLI invocations against the same workspace are last-write-wins (the file is small and the failure mode is “one passphrase lost, never data corruption”). Filesystem-level locking is on the v0.2 roadmap.
  • --passphrase-file takes precedence. When both flags are passed, the file wins and the keyring is not consulted.

​
Webhooks

Set CREWSHIP_BACKUP_WEBHOOK_URL (and CREWSHIP_BACKUP_WEBHOOK_SECRET) on the server process to receive a signed POST for each backup lifecycle event. Delivery is fire-and-forget from a goroutine — a slow or down webhook never blocks the backup run.
Each event is JSON with the shape:
Events: backup.created, backup.failed, backup.restored. Each request carries an X-Crewship-Signature: sha256=<hex> header — HMAC-SHA256 over the raw body using CREWSHIP_BACKUP_WEBHOOK_SECRET. Receivers must verify the signature (same scheme as Crewship’s inbound webhooks; validate via webhook.ValidateHMAC after stripping the sha256= prefix). The secret is required whenever URL is set — sending a body unsigned would let any network listener forge events to a downstream consumer that trusts the feed. URLs with userinfo or query strings are redacted before ever appearing in logs / audit rows, so basic-auth credentials or signed-URL tokens do not leak.

​
Metrics

GET /api/v1/admin/backups/metrics (instance administrators only) returns a point-in-time snapshot of process-lifetime counters. The numbers reset on restart — persistent observability belongs in the audit log and its dashboards.
Duration quantiles are approximated from an in-memory ring buffer — fine for the dozens-to-hundreds of samples a single host accumulates between restarts; not a general-purpose histogram. For long-horizon reporting, ingest the backup.* rows from audit_log.

​
Instance backup and recovery

An instance bundle is the whole Crewship in one file: a page-level snapshot of the entire database, every file store (attachment and workspace files, memory and its version history, page projects, chats, skills — each file listed with its SHA-256), and every workspace’s crew containers. It is the disaster-recovery path for a host: restore it offline into an empty data directory and boot a server from it. Only instance administrators can take, check or restore one: the CREWSHIP_OWNER_EMAIL owner, or anyone named an instance admin. An install that named nobody named the owners of its oldest workspace once, the first time anyone’s status was asked, and never again — removing every named admin later does not bring that rule back (crewship admin instance add-admin --local on the host is the way in). Every instance bundle is encrypted; age recipients are preferred over a passphrase.

​
The quiet window

Database and files must agree, so the copy is taken in a quiet window. It opens in three steps:
  1. Running work drains. The server waits for running agent work, started work items and routine runs to finish — up to two hours. A run that cannot drain is recorded as skipped, never forced.
  2. Writes drain. New writes are refused from here on, and the server waits for the writes already in progress to finish. A write is every mutating request (edits, uploads, sidecar and agent calls, inbound webhooks) and every background task that changes data: retention and garbage-collection sweeps, memory syncs and nightly consolidation, schedule fires, queued-work claims and notification re-deliveries. Reads never count, so open streams, live views and downloads do not hold the window up. The wait is bounded by the drain timeout — 60 seconds by default, set with CREWSHIP_BACKUP_DRAIN_TIMEOUT (a duration such as 90s). A write still running when it ends means the copy would not be consistent, so the window is released without copying and the run is recorded as skipped (“writes did not drain”), then retried like any other busy run.
  3. The copy runs. Only once no write is in progress does the copy start, so the database snapshot and the file copy share one boundary no write crosses.
While the window is open (steps 2 and 3), mutating requests answer 503 with Retry-After (the backup and instance-hold pages and sign-in stay usable and are not counted as writes), inbound webhooks answer 503, schedules and queued work do not fire, and a long sweep steps aside at its next batch and resumes after the window. A hard cap of 20 minutes on the copy releases the hold and abandons the copy; any error releases it too. All of this lives in memory only, so a crash mid-window comes back with nothing held. Packing and encryption run after the hold is released; the run reports how long writes were refused (hold_ms, the write drain plus the copy). Anything the bundle should hold and does not — attachment files missing from the storage root, memory versions without content, a provisioned crew whose container was gone, a vault key the server could not resolve — is recorded in contents.incomplete, never left out silently.

​
What the window guarantees, and what it does not

Covered — nothing below can write while the copy runs:
  • Routine runs. A run is admitted by the window before it writes anything (its idempotency reservation, its run row, its steps) and counts as running work from that moment until it returns — not only once its row says running. Admission is refused while the window is closing or held: a due schedule stays due, a deferred run stays pending and fires on the first sweep after the release, and any other start (a resume after an approval or a signal, a boot-time resume) waits for the release. A run admitted a moment before the window closed is still counted when the server checks running work again after the write drain, so the window does not hold and retries instead. A run started inside a request, or by another run (a nested routine), is admitted with its parent and counted the same way.
  • Queued work and scheduled agent work. A work item is starting from the moment it is claimed, and the claim itself is a write the drain waits for.
  • Agent runs started from a request (assignments, webhooks, queries) hold their workspace’s guard, which the window takes as its last step; a run that arrives after that is refused with “workspace is being backed up”.
  • Missions. Each pass of a mission’s loop is a write the drain waits for, and no pass runs while the window is open; an assignment a pass dispatches is admitted like a routine run.
Not covered:
  • Another server process. Admission and the run count live in this process’s memory. A second replica writing to the same database is only seen through its rows (running agents, routine runs and started work items), so its runs are still counted only once their row says so.
  • Work that ignores the gate. A code path that writes without entering the gate or being admitted — a mission’s timeout clean-up, for one — is not held back. The gate is enforced by review and by the guard tests, not by the database.

​
The recovery kit

Credentials and other sealed values are encrypted with vault keys (ENCRYPTION_KEY, ENCRYPTION_KEY_V2 …) that live in the environment, not the database, so a database backup alone restores credentials nobody can read. With the recovery kit on, every key version the database references rides inside each instance bundle’s encrypted payload — never in a workspace or crew bundle:
Whoever holds a private backup key can read every secret in the instance. Keep the age private key on a different device from the live ENCRYPTION_KEY.

​
Recovering an instance

crewship recover runs with no server: it verifies the checksum, decrypts, refuses a non-empty data directory (unless --force), lands the database and every file store and checks each file against the bundle’s index, applies migrations forward, drops every sign-in session, clears locks and leases, rotates the session signing key, and writes the kit’s keys to <data-dir>/recovered-keys.env (mode 0600) — or says plainly that credentials stay unreadable when the bundle has no kit. Its report (ok, partial with every gap, or failed) is printed and stored in the restored database’s restore history. The restored server boots with automations held: routines and schedules do not fire, inbound webhooks answer 503, queued work and notification retries do not start. Check the server, then resume each: crewship admin instance holds resume routines|webhooks|queue|all (audited). Crew container files are kept under <data-dir>/restore-staging/crews/ until the crews exist to receive them, and complete container environments under <data-dir>/restore-staging/environments/ until the server runs with Docker (landing them).

​
Proving a backup

Three proofs, never collapsed into one flag: A drill runs the same restore as crewship recover into a throwaway directory with nothing started — no message sent, no webhook called, no model run paid for — and reports ok, partial (what failed) or failed. --post records it on the server. crewship admin instance backups restore-checks <path> answers space, format, runtime, unsafe host settings and conflicts before a restore. See Instance-Scope Backup Security for the threat model and crypto chain.

​
Upload an encrypted archive

In Recovery → Upload a backup, choose a .tar.zst archive from another machine. The upload shows progress, can be cancelled, and accepts up to 64 GiB when sufficient disk space is available. The server validates the encrypted AGE envelope and payload checksum without receiving a private key. Uploading does not restore data or prove that its contents can be recovered. From the CLI, with an instance-admin token and no selected workspace:
The raw API is POST /api/v1/admin/instance/backups/bundles/upload with Content-Type: application/octet-stream. It returns the catalog path, scope, format and checksum proof. Re-uploading identical bytes preserves existing proof. Catalog changes and their instance audit record are one transaction. Older archives requiring conversion are marked explicitly. After upload, select the corresponding scope, check the contents and perform a test restore.

​
Persistent quota-service disks

Format v4 includes standalone quota-service disks, in addition to crew container files. The host quota helper must be configured; a missing transport rejects the backup instead of silently losing these disks. Failed capture retains service maintenance and requires an explicit operator retry. After verifying that the failed capture’s writers are stopped, retry crewship backup create with --recover-services. A backup retry cannot take over a restore fence; land recovered images before attempting a new backup. After whole-instance recovery, these disks are verified and staged with fresh generations. On the destination instance, first run crewship admin instance backups services land --dry-run, then crewship admin instance backups services land. Keep the source archive and staged images until the destination’s services and data have been checked.

​
Backup plans and scheduled runs

A backup plan tells the server what to back up, when, for how long and who can open the bundles. Plans are instance-level and belong to instance administrators (Admin › Backups › Schedules, or crewship admin instance backups plans). Every run happens in the server’s own backup service — never through an agent or a model.

​
Presets

A complete plan writes an instance bundle (see Instance backup and recovery): its runs drain running work, hold writes for the copy only, and carry the vault keys when the recovery kit is on. Its environment runs also capture every crew’s complete container environment.
After crewship recover, a server boots with its schedules held. Scheduled and catch-up backup runs are then recorded as skipped (“scheduled backups are held”) until crewship admin instance holds resume routines (or all): a scheduled run on a server nobody has checked yet would rotate the good pre-restore copies away. A manual run still starts, so you can back up the restored server whenever you choose.
A custom backup never counts as protecting a workspace: the Overview judges protection by full backups only, and a plan’s counts_as_protection says false. The categories and what each needs: A memory-only plan therefore carries the crews and agents rows (the dependency), the memory tables, the crew memory trees and the memory version blobs — no chats, no issues, no container working files. Issues, inbox items and approvals have no category of their own and ride full backups only. crewship admin instance backups plans preview --contents memory shows the resolution before you save. The bundle’s manifest records kind: custom and its categories, and a restore with --replace from a custom bundle is refused, because replacing would delete everything the bundle does not hold.

​
When plans run

  • Timezone. time_of_day is a wall-clock time in the plan’s IANA timezone, so 03:00 stays 03:00 across daylight-saving changes. Cadence is daily, weekly (weekday, 0 = Sunday), monthly (monthday, or the first Sunday when none is set) or custom with a five-field cron expression, parsed the same way routine schedules are.
  • Environments. With env_mode: complete, env_cadence picks the runs that also take complete environments: every run, weekly (Sundays) or monthly (the first Sunday); a weekly or monthly plan takes them with every run. Such a run (environments: true) captures each crew’s complete environment together with its files, and its environments phase says how many were captured and how much of their image layers is new.
  • One run per night, whatever happens. The scheduler looks every 30 seconds. It records each due run and the plan’s next run time in one transaction, and the database refuses a second run for the same plan, due time and workspace — a restart or a second server cannot run the same night twice.
  • Catch-up. If the server was down over one or more due times, or reached a due time more than ten minutes late, it runs one catch-up backup when it is back (trigger: catchup), never one per missed night. The catch-up takes environments if any missed night would have.
  • Busy workspaces. Before a run starts, the scheduler checks for agents at work and routine runs in progress in its workspace. While busy, the run waits (phase busy_wait) and looks again every busy_retry_minutes (15 by default), up to busy_wait_minutes (120) after it was due. Still busy then, it is recorded as skipped with the reason.
  • One at a time. One run executes at a time; the rest queue in order. A plan over several workspaces makes one run per workspace.
  • Retention. After a good run the plan’s keep rules apply to that plan’s own bundles — per workspace for a Workspace backup plan, and to the plan’s instance bundles as one group for a Complete recovery plan — keep_min newest copies are never aged out, then the newest copy of each of the last keep_daily days, keep_weekly weeks and keep_monthly months. The newest keep_min checked copies are kept too, and a copy that recorded gaps (incomplete) never counts as one of them. Whatever newer copies exist, retention never removes the newest complete copy whose test restore succeeded, nor the newest complete copy whose contents were checked — a string of incomplete runs cannot rotate away the last copy known to be whole, and those copies keep their off-site copies as well. Pinned bundles, manual bundles and other plans’ bundles are never touched. A bundle retention removes leaves the catalog, gives back the environment layers no other bundle needs, and loses its off-site copies.

​
Runs and their phases

Every run is a row with its phases, shown in Backup history and by crewship admin instance backups runs: A run ends done, incomplete (a bundle was written and its manifest names what it lacks), failed, skipped or interrupted. When the server stops in the middle of a run, the next start marks it interrupted, removes its partial bundle file and its staging (for an instance run, the staging directory and partial bundle too), and retries it once; the interrupted row gets retried_at when the retry finishes, and a retry that is itself interrupted is not retried again. A held instance (after crewship recover) retries nothing on its own, and a restored server never re-runs a backup that was in flight on the source. crewship admin instance backups run (or Back up now) starts a run at once. It answers with the run id immediately and the backup finishes on the server, so neither a closed browser tab nor the CLI’s 30-second timeout cancels it. Without a plan, the recipients come from the plan that covers the same scope, unless you give --recipient or --passphrase-file; a passphrase stays in the server’s memory until the run starts and is never stored. --wait follows the run to the end, and crewship admin instance backups run-status <id> shows one run. Every run — scheduled, catch-up or manual, instance or workspace — is the same kind of durable run, listed by backups runs.

​
Overview

crewship admin instance backups overview (Admin › Backups › Overview) answers whether a restore would work, for the whole instance or selected workspaces:
  • Verdict — the weakest proof among the latest full backups of the selection: verified (test restore passed), partial (a test restore, or the bundle itself, found gaps), failed, contents_checked, checksum_only, or none. The whole-instance view counts instance backups only, so it says none until the first one is written, and tells you how many workspaces have workspace backups.
  • Status rows — what is protected, how often, where, how long copies are kept, and the last test restore. The off-site copy is not counted until one has been uploaded and checked.
  • Needs attention — the latest backup’s gaps per workspace (missing attachment files, memory versions without content, crews without their container), workspaces never or not recently backed up (the plan’s stale_alert_hours, 36 by default), workspaces on no plan, and failed, skipped or late runs. Workspaces that hold no crew (every signup gets one of its own) are not judged one by one: one item says how many of them have no backup and names up to three; they get no coverage row, and the summary counts them apart (“N workspaces without crews”). The whole-instance view also says when the latest instance backup cannot unlock credentials on a new server — the vault key versions its sealed values use that its recovery kit does not carry (all of them when the kit was off), read from the bundle’s own manifest. When the disk is too full for the next run to start (see Limits and the space floor), the first item says so (“Backups will not start: the disk is 95% full”), with the figures, what to do, and a link to Storage.
  • Fourteen nights — one mark per night: ok, incomplete (a backup was created but recorded gaps — missing attachment files, a crew without its container…; drawn in the warning colour, never green), late (a catch-up), skipped, failed or no run, with the best proof reached that night.
  • Space — what backups use, the free and total space where they are kept, the staging a new backup needs (twice the largest recent bundle of the scope: the encrypted staging file and the finished bundle exist side by side until the bundle is renamed into place) and the room a restore needs (the largest bundle × 3: payloads compress about 3:1, and a restore stages the decrypted payload). space.min_free_percent is the floor in force and space.refusal why the next run would not start, or null.

​
Complete container environments

A backup’s crew files (the Quick, Standard and Full levels) are the chosen folders only: /workspace, the crew memory tree, /output, the home and tools volumes, and with Full /var/lib. Restoring them needs the crew’s image to still exist — and Crewship builds crew images locally (crewship-cache:*), so a lost host loses them. A plan or run set to Complete environment (Admin › Backups › Schedules › Container environments, env_mode: complete, --env-mode complete) also captures each crew container’s complete environment.

​
What is captured

Not captured: what was only in memory. A restored environment starts its processes fresh — a running shell, an open editor, an in-memory cache and any process state are gone. Services that keep data only in memory should declare a flush hook that writes it to disk. Secrets. Crewship never puts credentials in a crew container’s environment: they are written per run to /secrets (a tmpfs) and to that run’s exec environment, and neither is captured. Any environment variable whose name looks like a secret (…TOKEN, …SECRET, …PASSWORD, …API_KEY, …ACCESS_KEY, …CREDENTIAL, …AUTH…, …SESSION…) is left out of the configuration by name, and blanked in the committed image, whose config would otherwise carry the container’s whole environment. The dropped names are recorded; their values never are.

​
Not carried over for safety

These host-level settings are recorded, listed in the restore checks as Not carried over for safety, and restored switched off — turn each back on by hand after review:
  • a Docker socket mount (docker.sock, containerd.sock, podman.sock) — not mounted;
  • privileged mode;
  • host PID, network, IPC, UTS or user namespaces (host network falls back to the default bridge);
  • added capabilities (cap_add);
  • device mappings and GPU requests;
  • host path binds — the path is not bound again; its captured content comes back in a volume at the same place;
  • published host ports;
  • seccomp, apparmor or SELinux label profiles that switch confinement off.
The binds Crewship adds to every crew itself (the sidecar binary and the entrypoint from its runtime directory) are not taken from the backup either: Crewship supplies them when it creates the crew container.

​
Encryption of the layers

A layer is a copy of the container’s root filesystem and can hold private files, so the store never keeps one in the clear — on this server or off-site — and still stores a layer shared by many bundles only once:
  • The store works in generations. Each has a random 32-byte key.
  • Every layer is encrypted with a key derived from the generation key and the layer’s own SHA-256 (HKDF), as a chunked XChaCha20-Poly1305 stream. The result is deterministic, so the same layer under the same generation is the same encrypted object, stored once and named by the SHA-256 of the encrypted bytes. Neither the store, the manifest nor a bucket shows a layer’s own digest.
  • The generation key is sealed with age to the backup’s recipients, keys/<generation>/<set>.age beside the store, and every bundle that uses the generation carries the key inside its own encrypted payload (environment-keys/<generation>). Restoring a bundle, landing its environments after crewship recover, and a test restore (drill) need nothing but the backup’s identity; a drill also proves that every layer in the store opens with the key the bundle carries.
  • So that the next backup can add layers to the same generation, the server keeps the key too: in memory, and sealed with its vault key (ENCRYPTION_KEY) in keys/<generation>.json. Someone who copies the backups directory without the vault key reads no layer. Without a vault key the server keeps it in memory only, and after a restart the next backup starts a new generation (its layers are then stored again once).
Recipients and rotation. A backup whose recipients include everyone the current generation was ever sealed to keeps using it; a recipient added since gets its own sealed copy of the key, and the layers are not encrypted again. The trade-off: whoever can open a generation’s key can read every layer stored under it, including layers added after they were given it. So when a recipient is removed — a backup’s recipients no longer include everyone the generation was sealed to — the store starts a new generation: every layer from then on is under a key the removed recipient never had. What it could open before (older layers, older bundles) stays open to it; rotate the backup keys and let retention age those copies out if that matters. A backup encrypted with a passphrase, or one without recipients, always gets a generation of its own. Older stores. Layers captured before this encryption stay as they were until retention removes the bundles that need them; they restore as before, but they are never copied off-site (the copy fails and says so). Take a new backup to get encrypted layers.

​
Shared layers and retention

The store lives beside the bundles, in <backups dir>/environments/: blobs/sha256/<hex> holds each distinct encrypted object once, keys/ the generations and their sealed keys, index/ one record per captured environment. Nightly captures of the same crew share every base layer, so a new capture adds only what changed. The database tracks which bundle needs which blob (environment_blobs, bundle_environment_refs); when retention or crewship backup delete removes a bundle, its references go and only blobs no other bundle needs are deleted. A capture in progress holds its blobs under a pending reference until its bundle is written, so a concurrent cleanup never takes them. The bundle itself carries each crew’s record (environments/<crew>.json, encrypted with the rest of the payload) and the generation key, and its manifest lists the encrypted objects it needs (contents.environments, readable without the key so retention and off-site copies can work on it). With CREWSHIP_BACKUP_ENV_INLINE=1 new bundles carry the layers inside the payload instead (environment-blobs/): self-contained, and larger. A bundle copied off-site without inline layers takes its blobs along — see Environment layers off-site.

​
Restoring an environment

Before anything changes, the restore checks (crewship admin instance backups restore-checks, Recovery › Checks) add one runtime check per environment: The room the layers need counts toward the space check, and with a key the recorded unsafe settings are listed. A restore then loads each environment before its crew’s containers are looked for: the archive is reassembled from the store and loaded into Docker. For a Crewship crew the image also gets its crew image tag back when this server has none, so Crewship recreates the crew container from the captured environment, with the runtime settings it manages. A container Crewship does not manage is recreated with the sanitized configuration, its volumes get their captured content, and it is started. Each environment is reported restored, rebuilt or skipped with the reason (the restore’s environments field), and anything but restored makes the restore partial.

​
After an offline recover

crewship recover runs without a server and usually without Docker, so for an instance bundle it only stages the environments: each workspace’s records under <data-dir>/restore-staging/environments/<workspace>.tar.zst and the layers they need under .../environments/blobs/, copied from the store beside the bundle (or read from the bundle when it carries them inline). A missing layer is a warning in the recover report: that crew’s data comes back, its environment is rebuilt. Once the recovered server runs with Docker:
POST /api/v1/admin/instance/backups/environments/land loads every staged environment the same way a restore does, writes landed.json beside them, and records one instance audit entry. Crew files staged under restore-staging/crews/ are still landed from a workspace bundle once the crews run.

​
Limits and the space floor

crewship admin instance backups settings set (Admin › Backups › Storage) decides how hard a backup may push the server: Before a run starts, the server compares the room it needs (twice the largest recent bundle of the same scope) with the free space where bundles are kept. A run that would leave less than 10 % of the disk free does not start: it is recorded as skipped with the figures and what to do, and an incident is raised when it happens a second time in a row (see Incidents and alerts). The refusal is not silent anywhere: the run’s reason in Backup history, the first Needs attention item on the Overview (with a link to Storage), the Space card (“New backup runs will not start”) and crewship admin instance backups overview all say it. The floor is set by one environment variable on the server: The default stays 10 %, because a backup that fills the disk takes the server down with it. Lower it only on a host whose disk is shared with other data and is routinely fuller than that — a development box at 95 % refuses every run at the default. Set it where the server’s environment is set (the systemd unit or .env), restart the server, and the next run uses it:
The better fix is room: free space on the disk, keep fewer copies (the plan’s keep rules), or move copies off-site. Staging is encrypted before it touches disk. A workspace run’s payload streams through the compressor and the age encryption straight into one staging file, whose checksum and size are counted on the way. An instance run’s consistent copy, taken inside the quiet window, never waits on disk in the clear either: the database snapshot is held in memory (SQLite’s online backup API into an in-memory database) and serialized straight into the sealed payload, and every other staged file — each file store’s copy, the crew and environment archives — is written through a key generated for that run that exists only in the server’s memory (XChaCha20-Poly1305 in 64 KiB chunks) and decrypted while packing. The quiet window is no longer for it. Holding the snapshot costs memory: about the database’s size during the run, twice that for a moment while it is serialized. Staging is removed after the run, on failure, and at the next start after a crash.

​
Incidents and alerts

When a backup goes wrong, the people who run the server hear about it — the instance admins, never workspace owners who may not administer the instance. One incident per plan and kind: A single skipped run pages nobody: one night the server stayed busy, or the writes in progress did not drain in time, is shown as skipped with its reason in Backup history, as a skipped night in the Overview’s nights strip and as a Needs attention item — and the next night usually runs. The second skipped run in a row raises the incident (set CREWSHIP_BACKUP_SKIP_ALERT_AFTER on the server, 1–30, to change how many; 1 pages on every skip), and a plan that keeps skipping past stale_alert_hours also raises stale. A good run resolves both and starts the count over. A plan over several workspaces runs once per workspace for one due time (or one Back up now). Its failed and stale incidents clear only once every run of that group has finished and none failed, was interrupted or was skipped — one workspace coming back well never hides another one failing. A retry after an interruption stands in for the run it retried. The reminder’s message names the command to run, for example “No test restore in 32 days — run crewship backup drill --bundle <newest bundle> --identity <key file>.” It is delivered once when it opens, like stale, and follows the drill switch under Keys & alerts. A repeat raises the incident’s count instead of opening another. Each instance admin gets one inbox item per incident, in their primary workspace (the oldest they own or administer, else the oldest they belong to) and visible to them alone: “Backup needs attention · Complete recovery backup failed. Last successful backup: 32 hours ago.” with View failure and Retry. A repeat refreshes the same item and marks it unread; resolving the incident resolves the item. The item routes under the system.health notification category, so each admin’s own channels — email, Slack, Telegram and the rest, per their notification preferences (Settings › Notifications) — carry it too. An instance admin who belongs to no workspace has no inbox; they see incidents under Admin › Backups and with crewship admin instance backups incidents. Which kinds are delivered is chosen under Keys & alerts (--alerts); the others are still recorded.

​
Alert channels

Beside the inboxes, the Alerts card’s Tell row lists the notification channels that can carry backup alerts — Slack, Discord, Teams, e-mail, signed webhooks, Opsgenie and the rest of the provider catalog — and every one ticked hears every delivered incident, whoever is or is not looking at an inbox. The channels are the ones already configured under a workspace’s Settings › Notifications; nothing is stored twice. A channel is offered when it is workspace-wide (never a member’s personal channel), switched on, its category allowlist admits System health, and its provider is switched on in Admin › Notifications. With none, the card says to configure one there. Each channel gets one message per incident state change: when the incident opens, on each repeat, and a “resolved” message when it closes (only to the channels that were told it opened). The message reads:
The link is made absolute against CREWSHIP_PUBLIC_URL; without it the message carries the path alone. Delivery is the notification system’s own: the channel’s sealed secret, the SSRF-safe transport that refuses private and loopback addresses at connect time, the provider kill switch and the secret scrubber. Every attempt is a row in the notification delivery log (visible under that workspace’s deliveries), deduplicated per channel and state change, so a retry never sends the same change twice. A failed send is recorded with its reason, retried by the delivery recovery sweep (every 2 minutes, up to 5 attempts) and shown on Overview as “Backup alerts could not be delivered to <channel>” until a later alert gets through. A retry is dropped, with the reason in the log, when the incident resolved in the meantime or the channel left the route or was switched off. Send test (crewship admin instance backups test-alert --channel <id>) sends a test message through the same path and says whether it arrived; tests are not written to the log. A channel already on the route that later becomes unusable (switched off, deleted) stays on it — other settings still save — and its alerts fail with the reason until it is removed. Adding a channel checks it is usable now. When the server itself is down its inbox is down too, so set a heartbeat: a public https URL (a healthchecks-style service) the server pings once a plan’s run comes back good — for a plan over several workspaces, once every run of that due time (or manual start) has finished and all of them are done — with a 10-second timeout and one retry, never delaying the run. A ping says that plan’s latest round succeeded; it is not a health check of every plan. The outside service alerts when the pings stop. The last ping’s time and outcome show in the settings.

​
Backup keys and the recovery sheet

Backups are encrypted to backup keys: age public keys registered with crewship admin instance backups recipients add (Keys & alerts). Plans name them in recipient_ids. The private halves never reach the server; keep two keys held by different people, so one lost key is not a lost backup. A key a plan still uses cannot be removed. crewship admin instance backups recovery-sheet (the Recovery sheet download under Keys & alerts) prints a Markdown page to keep off the server: where the bundles are, the keys’ names and fingerprints, whether the vault keys ride in the recovery kit, the exact crewship recover and crewship backup drill commands, the plans and the instance admins. It holds no secret.

​
Admin UI

Admin › Data › Backups is for instance administrators. It is a page of its own, /admin/backups, with a side panel like Admin › Security: the scope, Overview, the runs by result, the plans, Recovery and the instance settings. Each page is deep-linkable as /admin/backups?section=<page> (an old /admin?tab=backups&section=<page> link still lands there): The panel’s Scope chooses Whole instance or Selected workspaces; the choice is kept in the URL. Admin › Data › Data retention sets how long each kind of live data is kept per workspace, and shows a dry run of what a change overwrites before it is saved.

​
Recovery in the console

  • Whole instance (Empty server, Isolated instance). The server keeps no private key and has no online instance restore, so after Keys and Checks the wizard shows the command to run where the key is, for the bundle picked, with a Copy button — no dry run in between. Empty server: crewship recover --bundle <bundle> --identity <key-file> --data-dir /var/lib/crewship on the new host. Isolated instance: crewship backup drill --bundle <bundle> --identity <key-file> --post, a test restore into a throwaway directory whose result is recorded on this server. Replace <key-file> with the file holding the private backup key.
  • A workspace or one crew. Checks, a dry run with its full report, then the restore, through POST /api/v1/admin/backups/restore for the workspace the backup belongs to (or the one ticked in the strip). An instance admin need not be a member of it.
  • From off-site storage. Step 1’s From off-site storage… lists the bundles a destination holds; Fetch downloads one back to this server (see Restore from an off-site copy) and the wizard carries on with it, and Use takes a copy already here.
Pages whose server endpoints have not shipped yet say “Not available on this server yet” rather than showing figures. Overview, Backup history and Schedules read the plans, runs and overview endpoints described in Backup plans and scheduled runs; checks, downloads and workspace restores go through the existing backup endpoints.

​
Off-site destinations

A backup that exists only on the server it protects does not survive losing that server. Off-site destinations copy finished bundles to object storage. The first destination type is S3-compatible storage: AWS S3, Cloudflare R2, Backblaze B2 (S3 API), Wasabi and MinIO. Google Drive is planned later. How a copy is made:
  • Only finished bundles move. Bundles are encrypted before they leave the server; the store never sees plaintext or keys.
  • A copy counts only after its stored bytes are proven. Every upload sends the SHA-256 of each request body in x-amz-checksum-sha256, so the store refuses bytes that changed on the way. After the upload the server asks the store for its own SHA-256 checksum of the object (HEAD with checksum mode enabled; for a multipart upload the composite checksum-of-part-checksums) and compares it with the one computed from the local bundle. A store that returns no checksum (some S3-compatible stores ignore the headers) has the copy downloaded and re-hashed instead — the object’s size and the uploader’s own x-amz-meta-sha256 metadata are checked too, but never count as proof on their own. Which proof was used is recorded per copy (verified_by: provider_checksum or download_rehash) and shown under Storage (“verified by the provider’s checksum” / “downloaded and re-hashed”) and in the run’s off-site phase. A copy that fails the check is deleted and reported, never counted.
  • Large bundles upload in 64 MiB parts. A failed or cancelled upload is aborted so no half-written parts stay billed in the bucket.
  • Transient errors retry. Server errors, throttling and timeouts retry with backoff; permission errors do not.
  • Uploads respect the upload limit set under Storage, so a nightly copy does not saturate the server’s link.
  • Connection test. Saving a destination writes, checks and deletes one small object first, and nothing is stored when that fails; the error names the step that failed (upload, check or delete) so a missing bucket permission is obvious.
  • Retention reaches the bucket. When a plan’s keep rules remove a local bundle — workspace or instance — its off-site copies are deleted with it.
Add a destination under Admin › Backups › Storage (Add S3-compatible storage) or with the CLI, then name it in a plan’s destinations (Schedules › Where):
Instance bundles land under instance/ and workspace bundles under workspaces/<workspace id>/ below the prefix. The Overview’s Where row names the destinations, and the “Local copy only” warning disappears once every recent backup of the selection has a verified off-site copy. A failed upload leaves the local bundle as it is, marks the run’s off-site phase failed and raises an offsite incident.

​
Environment layers off-site

A bundle whose complete container environments keep their layers in the shared store (the default, not CREWSHIP_BACKUP_ENV_INLINE=1) would be missing them off-site on its own, so the copy takes them along:
The refs object is written first, then every layer the destination does not already hold with the same size and checksum (a nightly bundle usually uploads only what changed), then the generation’s sealed key files under <prefix>/environments/keys/<generation>/, then the bundle — each checked like the bundle. The layers go up exactly as the store holds them, encrypted; a layer stored in the clear by an older version is refused rather than uploaded. A layer missing from the local store fails the copy instead of leaving a bundle off-site that could not restore its environments. The run’s off-site phase says how many layers were uploaded and how many were already there. The destination is its own reference count: when retention deletes a remote bundle, it deletes the layers that bundle needed only if no other refs object at the destination still names them, so a copy survives even when the server’s database is lost. A restore from off-site downloads the bundle and the layers its refs object names into the backups directory and the environment store, checks each against its digest, and records the layers as the bundle’s, so local retention keeps them.

​
Restore from an off-site copy

When the server lost its bundles, or a new server is brought up from a bucket, fetch a copy back first; it is then an ordinary local bundle for a workspace restore, a drill or crewship recover.
copies list shows every bundle at the destination (not its layers or refs objects) and where this server already has it. copies fetch downloads the bundle and the layers its refs object names into the backups directory, each checked against its SHA-256, and records the layers as the bundle’s. A fetch can take far longer than one HTTP request may, so the server runs it as a job with its own six-hour deadline and answers at once; --wait (or Recovery in the console) follows it, and copies status <id> looks it up later. A server restart forgets its fetches; a half-downloaded bundle never appears. A bundle already on this server is refused with its path. API: GET /api/v1/admin/instance/backups/copies?destination=, POST /api/v1/admin/instance/backups/copies/fetch and GET /api/v1/admin/instance/backups/copies/fetch/{id} (instance admins).

​
Compatibility

The promise: every supported backup remains recoverable through the current recovery tooling, directly or through maintained converters. Precisely:
  • Supported means every bundle Crewship has ever written, from format_version 1 on. The oldest recoverable version is 1 and it is never raised.
  • Directly: the current restore, verify and inspect read the current format_version and the two before it (today 2 to 4).
  • Through converters: crewship backup convert turns a bundle from any older version into one at the current version, one step per version. When restore, verify or inspect meet a bundle older than they read directly, they print the exact convert command for it instead of failing with nothing to do.
  • Recoverable means everything the bundle actually contains comes back. Data an older version never collected cannot be invented; the converter names it in its report, and the converted manifest keeps the list.
  • A converted bundle is a new file. The original is only read, so it can be kept as it is.
What each step does today: The report also warns when a bundle predates per-table row counts (its completeness cannot be verified; the converter does not invent counts) and when it has memory-version rows but no memory-blobs section. Keys: converting needs the key that opens the bundle, --identity or --passphrase-file. The converted bundle is sealed to the same age recipients, or the same passphrase, unless --recipient names new ones. A legacy unencrypted bundle is converted only with a --recipient, because new bundles are always encrypted. The payload is decrypted as a stream and never written to disk in plaintext. How it is kept true: CI restores a small fixture bundle for every format version on every run, converting it first when it is older than current, and checks that the workspace, crew, agent, issue and comment, journal entry, credential and memory version come back and that the crew file sections are in the bundle for the docker phase to land. A change to the bundle format cannot merge without a converter step from the previous version and a fixture for the new one; tests fail until both exist.

​
Known caveats (v0.2 roadmap)

The following items are intentionally deferred to v0.2. They do not block production use but are worth knowing.
  • preBackup / postBackup hooks — no general user-defined hooks yet. A crew can declare a pre-pause flush command with the container label crewship.backup.flush; it runs only for backups that take complete container environments. Otherwise run an app-level flush manually before crewship backup create.
  • Remote backends — S3-compatible storage only (see Off-site destinations); Google Drive and GCS come later. For anything else use scp / rclone / restic, or stream a single bundle via GET /api/v1/admin/backups/download.
  • Scheduled backups — plans run on the server’s scheduler (see Backup plans and scheduled runs); the per-workspace crewship backup create still fits cron or systemd.timer.
  • Forward migration replay hooks. The plumbing is wired — migrations can register a per-version restoreBackfill function and the restorer walks the applied ∖ manifest set in ascending order after the main transaction commits — but no migration registers a hook yet. A failed backfill surfaces as ErrRestoreBackfillFailed; the restored rows are visible but may be missing backfilled columns until an admin investigates.
  • Cross-process keyring lock. The per-process mutex does not cover two concurrent CLI invocations racing on the same keyring file.

​
Common pitfalls

Losing the passphrase = losing the data. AGE bundles are not recoverable without the passphrase (or X25519 private key). There is no master key, no support escape hatch. Store the passphrase in a separate trust zone from the bundle — a password manager on a different host, a sealed envelope, anything that doesn’t share a failure mode with the disk holding the .tar.zst.
The advisory lock exists precisely because two concurrent backups race on the docker pause / docker unpause sequence. Only ever clear a lock when you can prove its holder is dead (CLI session crashed, host rebooted) and the 1-hour TTL hasn’t fired yet. When in doubt, wait for the TTL.
Like the Admin CLI trap — if the server runs with a custom data dir but you invoke crewship backup … without the same env var, the CLI defaults to ~/.crewship and operates on an empty/separate database. Backups succeed but capture nothing useful; restores produce “workspace doesn’t exist” errors. Export the same CREWSHIP_DATA_DIR the server uses (there is no --data-dir flag — the env var is the only override).
Restore takes the advisory lock but a running crewshipd may still hold open handles on tables the restorer wants to truncate. Symptoms range from “database is locked” SQLite errors to a half-applied restore that leaves the workspace in a non-bootable state. The lock guards against concurrent backups, not against the server’s own writes — stop the service first.
A bundle written by format v5 will restore on v5, v6, and v7 servers; v8 onward, the compatibility window has rolled past it. If you’re restoring from cold storage that’s been sitting for a year+, verify the manifest’s format_version against the destination first with crewship backup inspect.
Passing both raises a CLI error rather than silently picking one, and --no-encrypt is refused outright — every new bundle is encrypted. A bundle encrypted with --recipient age1… will not decrypt with a passphrase, and vice versa — match the restore flag to the original create flag.
crewship backup restore will not overwrite an existing workspace or crew under the same slug. If the existing one is stale and you want to replace it, delete it first (a crew via crewship crew delete; a workspace via crewship workspace delete <slug> --confirm <slug> — OWNER only, refused if it is your only workspace, see the workspace CLI); if you want both side-by-side, use --as-workspace=<new-slug> and then crewship crew start.
Peak RAM is fixed by the zstd window, but the extraction scratch directory at $TMPDIR/crewship-backup-* needs ~2× the bundle size on disk. Restoring a 4 GB bundle onto a host with 6 GB free in /tmp will fail mid-stream. Mount /tmp on the data volume or set TMPDIR to a larger filesystem.
A non-TTY second invocation will block waiting for stdin. Either pass --passphrase-file, pipe the passphrase via stdin redirect, or use --use-keyring so the first call seeds the cache.
User code in /workspace, the crew and per-agent memory trees under /crew, anything mounted into the agent container — all included in bundles at format_version 3 and above (see the memory warning under Bundle layout for what v1 and v2 bundles actually contain). Workspace-scope bundles also carry the content-addressed memory-version blob store ({MemoryRoot}/versions) behind memory_versions rows — see Memory-version blobs. AGE encryption is the only defence; treat a .tar.zst like any other backup of source code and credentials.
This is load-bearing — agents pull credentials from /secrets at runtime via Keeper, and including them in a portable bundle would defeat the SECRET-tier guarantee. Don’t add code that bypasses the secret-skipping filter without an equally rigorous out-of-band channel.

​
Security notes

  • The /secrets mount is never included in a bundle, and vault secrets stay sealed inside it.
  • Agent home folders are included, and they can hold credentials the vault never saw — ~/.aws, ~/.ssh, CLI tool logins. That is why every new bundle is encrypted and --no-encrypt is refused.
  • The workspace bind mount (user code), the per-crew and crew-shared memory trees under /crew (v3 bundles and above), and the workspace-tier memory-version blob store (memory_versions content, workspace scope only) are all included. AGE encryption of the payload is the only defence against leakage from physical bundle distribution — treat bundles like any other backup of your source tree.
  • Every backup.* event writes to audit_log with user, role, scope, sealed-payload SHA-256, and size. Dry-run restores write backup.restore.dry_run so auditors can tell rehearsals from real cutover.

​
Automatic pre-migration snapshots

Distinct from the manual crewship backup bundle system above: Crewship auto-snapshots the SQLite DB before any pending migration runs. This is the “binary upgrade went sideways, give me my data back” safety net.

​
How it works

Every crewship start calls database.SnapshotBeforeMigrate before database.Migrate. If any migrations are pending (rows in the migrations[] slice with version
MAX(version) FROM _migrations), the function:
  1. Resolves the SQLite file path from the connection DSN.
  2. Computes a target name: <dbpath>.pre-migrate-v<from>-to-v<to>-<UTC-RFC3339>.bak.
  3. Issues VACUUM INTO '<target>' — SQLite’s hot-copy mechanism. Safer than a plain file copy because it serializes against concurrent writers and produces a defragmented, WAL-checkpointed snapshot.
  4. Chmods the snapshot to 0600.
  5. Prunes older snapshots, keeping the 10 most recent per database file.
On error the boot aborts before any migration runs — silently continuing without a rollback point would defeat the entire purpose.

​
Opting out

Skips the snapshot. Useful for CI environments where the DB is ephemeral and snapshot I/O is just overhead.

​
Recovering from a bad migration

If crewship start succeeds, applies a migration, then exhibits runtime errors that point at schema drift:
1

Find the most recent pre-migration snapshot

2

Stop crewship

3

Replace the current DB with the snapshot

4

Run with the PREVIOUS binary (the one that wrote the snapshot)

The snapshot is a complete SQLite file — no special tooling needed. Tools like sqlite3 open it directly for inspection.

​
Limits of pre-migration snapshots vs manual backups

Use both. Pre-migration snapshots cover binary upgrades; crewship backup covers DR, host migration, legal hold, and forensic preservation. They are not interchangeable.

​
Data retention

Backups keep what existed when they were taken; retention decides how long the live database keeps it. Each workspace has nine windows, in days or forever, and a daily sweep deletes what falls outside them: Open work is never swept: an unread or blocking inbox item, a pending or escalated Keeper request, a request backing a live credential lease, and a chat with a delegation, an unanswered escalation, an unresolved reply, a pending approval, an unfinished routine run or a recent access grant all stay, however old. An instance administrator sets the windows for one, several or all workspaces in Admin › Data retention or with crewship admin instance retention set. Before anything is saved, the preview says how many rows the next sweep would delete. “All workspaces” means every workspace that exists now; what a workspace created later starts with is a separate setting (crewship admin instance retention defaults set) that touches no existing data. A save is held to the preview it confirmed. The console keeps the preview’s targets and changes as they were; changing the selection or a value drops the preview (an answer that arrives for an earlier selection is ignored), and Apply sends exactly what was previewed with the server’s preview_id as expect_preview. If a workspace or a window changed in between — another admin saved, or a workspace was created while “all” was ticked — the server answers 409, nothing is written, and the console says “Something changed since the preview — review again”. The defaults save works the same way, and so does crewship admin instance retention set and retention defaults set. Saving deletes nothing by itself; the next sweep does, and bundles taken before still hold what it removes. So a shorter window is safe to try only if your backups reach back far enough. The server also applies fixed limits nobody sets, such as journal compaction after 30 days, webhook receipts, delivered work bodies and page build history. crewship admin instance retention get lists them. See the Admin API reference for every window’s exact rule.
  • Backup API reference — REST endpoint shapes for every crewship backup subcommand.
  • Admin CLI — the host-shell write surface that backups protect against; always crewship backup create before running admin writes in prod.
  • Migrations Catalog — what migration ran when, and which restoreBackfill hook (if any) will fire on a cross-version restore.
  • Troubleshooting — recovering from a stuck migration or corrupt DB.
  • Security → Audit log — where backup.created, backup.restored, and backup.restore.dry_run events land.