Skip to main content

​
Backup & Restore

​
Overview

Crewship’s backup system produces portable .tar.zst bundles that capture a workspace (or a single crew) in one file. Bundles are AGE-encrypted by default, carry a versioned manifest, and can be restored on any Crewship instance that speaks the same format version (N-2 compatibility guarantee). The whole subsystem is admin-only by design: every backup subcommand requires the OWNER or ADMIN role on the workspace, and the runner refuses MEMBER / VIEWER calls at both the CLI parsing layer and the server-side handler — defence in depth, not just a UI veneer. The architectural choice that shapes everything else is “one bundle, one file”. Backups don’t depend on an external object store, don’t require a sidecar, and don’t shard across files. A .tar.zst is a single artefact an operator can scp to a backup host, hand to a customer for legal hold, or check into a private bucket — without the rest of Crewship being available. Inside the tarball, the manifest is plaintext JSON (so crewship backup inspect can read it without the AGE recipient key) and the payload is the encrypted SQLite snapshot plus any referenced workspace files. The forward-compatible manifest schema means a bundle produced on N can restore on N-1 and N-2; older bundles run their migration’s restoreBackfill hook so columns added since the snapshot are populated sanely instead of left at SQL defaults. Restores are intentionally cautious. --dry-run walks the entire restore plan — schema diff, row counts, blob deltas — and prints what would happen without writing a single byte to the destination database. Advisory locking (backup_locks table, per-workspace) prevents two concurrent restore runs from corrupting each other, and the lock file records the host + PID so crewship backup status can tell you who’s holding it. If a host crashes mid-restore, crewship backup unlock is the manual recovery — admin-only with a confirmation prompt, because clearing a real lock from another live process is how you trash a workspace.

​
When to use it

Backups are cheap to make and expensive to wish you’d made. The five canonical reasons to run crewship backup create:
  • Before any destructive admin write. Before crewship admin reset-password, before a database migration on a binary upgrade, before a large schema-changing PR lands in prod — capture the current state first. The bundle is the one-command rollback if anything goes wrong.
  • Disaster recovery / hot spare. Schedule a nightly crewship backup create --scope=workspace --passphrase-file … cron and ship the bundle to a separate host. If the primary disk dies, restoring onto a fresh binary is one crewship backup restore away — no replication agent, no streaming WAL, no extra moving parts.
  • Workspace migration to another host. Moving a workspace from a dev VM to a prod host (or between two prod hosts) is exactly what bundles are for. Create on the source, scp the file, crewship backup restore --as-workspace <new-slug> on the destination, start the crews, then crewship backup restore … --files-only to land their filesystem state. The --as-workspace rename avoids the “two acme workspaces colliding” hazard; the third step is not optional — without it the rows migrate and every crew’s workspace files and memory tree stay behind.
  • Legal hold or compliance archive. Customer leaves; you need to keep their workspace state on cold storage for N years. One AGE-encrypted .tar.zst is a forever-readable artefact — no live database, no service required. inspect later proves the bundle’s contents without decrypting.
  • Forensic snapshot before incident response. Suspected compromise of an admin account, or a “what was the state when X happened” investigation. backup create + immediate offsite copy preserves the audit trail before anyone (you, the attacker, the well-meaning oncall) starts changing things.
Skip backups for ephemeral development workspaces (the dev VM rebuilds them from seed scripts anyway) and single-agent throwaways with no user-visible state worth preserving — the bundle metadata cost is real (~few MB minimum) and not every workspace earns it.

​
Key concepts

​
Usage

The whole backup surface is the crewship backup command group — twelve subcommands that cover the create → verify → restore → rotate lifecycle (plus metrics, download, and self-test). The core nine are tabulated below with full flag reference and copy-pasteable examples; metrics is covered under Metrics, download under Known caveats. This table is the entry point. The minimum end-to-end loop is four commands: create to produce the bundle, verify to prove it’s not corrupt, restore --dry-run to prove the destination will accept it, then restore for real. Every other subcommand exists for retention (rotate), introspection (list, inspect, status), or recovery (unlock).
create and restore accept --use-keyring to cache and reuse the workspace passphrase via ~/.crewship/backup-keyring.enc. See Passphrase keyring below.

​
Bundle layout

  • Scope: workspace (workspace row + all its crews) or crew (single crew + its agents).
  • Encryption: AGE passphrase (default) or AGE X25519 recipient. --no-encrypt produces a plaintext payload for test / CI use.
  • Default location: ~/.crewship/backups/ on the server, mode 0700.
  • Naming: crewship-<scope>-<slug>-<iso-ts>.tar.zst. Collisions append -<hash8>.
Inside payload.age, bundles carry three distinct notions of “memory” — don’t confuse them:
  • Per-crew agent and crew-shared memory (crew/<slug> section) — the real memory tree: /crew/shared/.memory and /crew/agents/<agent-slug>/.memory inside the crew container. This is what an agent accumulates and what is hardest to regenerate. Collected by the docker phase, at every scope level including quick.
  • Per-crew declared output (memory/<slug> section) — the container’s /output mount. The section name is historical; it has never held agent memory.
  • Workspace-tier memory-version blobs (memory-blobs/<sha[:2]>/<sha> section) — the content-addressed audit trail behind memory_versions rows (consolidation output, HITL-approved proposals, pin snapshots). Collected once per workspace scope from {MemoryRoot}/versions, keyed by the sha256 the DB dump’s memory_versions rows reference. Restore rewrites payload_ref to the target’s blob root — see the Memory-version blobs row under Key concepts above.
Bundles at format_version 1 or 2 contain no agent or crew-shared memory, and their manifest says memory_included: true anyway. Up to and including v2, the collector read /output and filed it under the memory/ section, and the flag was set from “this crew had a container” rather than from anything that was actually captured. Everything else in such a bundle — workspace files, named volumes, DB rows, memory-version blobs — is real and restores normally, and the restore prints a warning naming the crews whose memory is not in there. Memory from before your first v3 backup cannot be recovered from an older bundle. Take a fresh backup after upgrading.
From format_version 3 onward every per-crew inclusion flag in the manifest is derived from what the collector observed itself writing, not from what it expected to write. A section that captured nothing reports false, which is also what stops the restore preflight from demanding a provisioned container for a crew that has no data.

​
Which rows a bundle carries

A workspace bundle is not “the whole database minus other workspaces” — it is one SELECT … WHERE … per table, and the WHERE is derived, not hand-written. DiscoverScopedTables walks the schema’s foreign keys backwards from workspaces and records, for each table, the chain of hops that ties one of its rows to a workspace. crews is one hop (workspace_id); agents is two (crew_id → crews.workspace_id); agent_skills is three. Most tables have more than one such chain, and which one is chosen decides which rows exist as far as the bundle is concerned. Two rules govern the choice:
  • A NOT NULL chain beats a shorter nullable one. A filter on a nullable column silently omits every row where that column is NULL — and for a column like “the run that produced this” or “the agent who claimed this task”, NULL is the ordinary state, not the exception. The walk therefore makes a first pass that follows only NOT NULL foreign keys, so a three-hop total path is preferred over a one-hop lossy one. A table that has no NOT NULL route out at all needs an explicit filter in workspaceFilterSQL, and TestScopedFilters_NeverTraverseANullableFK fails the build until it gets one.
  • Ties are broken deterministically. Two parents the same distance from workspaces used to race through Go map iteration order, so the same schema could produce different filters on consecutive runs. The walk is now level-synchronous over a sorted adjacency, so a bundle’s contents depend on the schema and nothing else.
The practical consequence for an operator: a table added by a migration is picked up automatically, but only once it declares an intent in BackupTableIntent. Until then, restore-time drift detection refuses rather than guessing — see migrations. memory_included counts files inside a .memory directory, not tar entries. The distinction matters because the container runtime creates crews/<id>/shared and crews/<id>/agents when a crew container is created, so counting entries would report memory for every provisioned crew whether or not an agent had ever written a note. crew_files_included is the separate flag for “the /crew section carries something” — init.sh and other crew-level content — and is what decides whether the section is restored at all.

​
Create a workspace backup

The CLI prompts twice for a passphrase (to guard against typos) and confirms success with a row summarising scope, size, format version, and the SHA-256 of the sealed payload.

​
Non-interactive / CI

Supply the passphrase from a file:
Or pipe a single line on stdin (falls back automatically when stdin is not a TTY and --passphrase-file is not set).

​
Asymmetric encryption

If the restoring party holds an AGE X25519 keypair, pass their public key instead of a shared secret:
--recipient, --passphrase-file, and --no-encrypt are mutually exclusive.

​
Back up a single crew

--crew accepts either a slug or a crew ID. Crew-scope bundles restore independently of their parent workspace.

​
List, inspect, verify

inspect only reads the plaintext MANIFEST — it never touches the sealed payload, so no passphrase is needed. verify recomputes the SHA-256 of the sealed bytes against the manifest and fails if the bundle was truncated or tampered with. Neither decrypts.
verify proves integrity, not completeness. It answers “are these the bytes we wrote?” — it cannot answer “are these all the rows there were”, because the manifest records no per-table row counts to compare against. A bundle that was written short reports ✓ VALID. The guard against writing one short is on the create side (see Which rows a bundle carries), not here.

​
Restore

The server rejects the restore if a workspace (or crew) with the same slug already exists. Override with --as-workspace <new-slug> or --as-crew <new-slug> to land the payload under a fresh identity.

​
Restoring under a new identity takes three steps

A rewritten restore forks the bundle’s rows under a new workspace id and slug. It cannot land container state in the same call, because the crews it just created have no containers yet. So the flow is:
--files-only restores the per-crew filesystem sections (workspace, memory tree, named volumes, /var/lib) and changes no database rows. It is authorised by provenance, not by the flag: step 1 records which bundle the new workspace was forked from — matched on the bundle’s payload digest — and step 3 is permitted only for a workspace that record covers. A bundle that was never restored into your current workspace is refused exactly as it was before. That provenance authorises only --files-only. Running the plain restore against a forked workspace is refused, and deliberately so: without a rewrite flag it takes the ordinary docker phase, whose crew identities come from the manifest — on a same-instance restore those are the source crews, and it would overwrite their live workspace and agent memory with the older backup while INSERT OR IGNORE kept the row counts looking untroubled.
Step 3 is what earlier versions described as “re-run restore without the rewrite flag”. That never worked and could not: the forked workspace’s id and slug match neither of the bundle’s, so the tenant guard rejected it every time. --files-only is the step that replaces it. It also addresses the crews the fork created rather than the crews the bundle names — the manifest still carries the source crew’s slug, so on a same-instance restore the older advice would have written a sibling’s backup over the source crew’s live data.
Restoring under the ORIGINAL identity (same workspace, or a genuinely empty instance) is unaffected: the docker phase runs in the same call and there is no step 3.

​
The target crew must be running

Every section is written through tar inside the container now (see Bundle layout), so a restore needs the crew container running, not merely created. A stopped crew is refused in the preflight, before anything is written, and the message names the fix:
crew start is the command for this: it starts the container, and provisions its image first if the crew has none — so it covers both the stopped crew and the never-provisioned one. It is not crew provision. Provision builds an image and starts nothing; on a cache hit it reports provisioned in seconds while the container stays stopped. This message used to name it, which sent operators round the loop once more before they got here. The preflight also refuses when a path the bundle writes to cannot be written by the agent — a directory or file left owned by root or by the memory sidecar. That is a refusal with nothing written, which is the point: the same restore used to fail partway through, after earlier sections had already landed. Hand the named path to the agent (chown -R 1001:1001 <path>) and re-run. Paths the bundle never touches are ignored, so unrelated root-owned files under /home/agent do not block anything.
One thing is reported without failing the restore: if the crew memory tree’s group and setgid bits cannot be fully re-applied afterwards — a .memory directory owned by the memory sidecar, say — the restore succeeds and warns, naming the directory. The data is on disk; only the sharing contract is degraded, and failing at that point would roll back the database while leaving the container’s files in place.

​
Dry run

A dry-run decrypts the bundle, validates the manifest, replays the DB transaction, and then rolls back. The only side effect is a single backup.restore.dry_run row in the audit log — handy for proving a bundle is restorable before the real cutover. The distinct audit action lets auditors tell “verified” apart from “actually restored”.

​
Credential security levels are re-checked on the way in

A restore writes the bundle’s rows as it finds them — with one exception. credentials.security_level is not data, it is the input to a Keeper gate, so every value is checked against the tier table (L1–L4) as the bundle is applied. A level the table does not define — 0, a negative number, a tier from a future release, a corrupt or tampered value — is written at the strictest tier instead:
Three things about that behaviour are deliberate:
  • The credential is kept, not dropped. Recovering an instance that is quietly missing a secret is a worse outcome than recovering one credential at too strict a tier.
  • The clamp goes up, never down. The strictest tier is what Keeper already resolves an unrecognised level to at every decision point, so no gate changes behaviour; what changes is that the stored value now agrees with the gate.
  • You are told either way. The count and the affected credentials come back in the API response, and a credential.security_level.clamped entry lands in the journal with the value the bundle carried — so the question “why is this credential L4?” has an answer months later.
--dry-run reports the same thing (as would be clamped) without writing, which is the cheapest way to check a bundle before a cutover.

​
Delete & rotate

rotate applies retention per workspace — it never touches another workspace’s bundles. Either --keep-last N (bundles above the N newest are dropped) or --keep-days D (bundles older than D days are dropped) must be positive; both can be combined. delete requires interactive confirmation, or --force in scripts / CI. The same rule applies to backup unlock.

​
Lock semantics

Each workspace holds at most one advisory backup lock at a time (table backup_locks, per-workspace PK). The lock:
  • Is taken before any DB dump or docker pause and released by a deferred Release() on the happy path.
  • Has a 1-hour TTL (DefaultLockTTL); a crashed backup self-heals after the window.
  • Blocks concurrent backup create calls — the second caller gets HTTP 409 Conflict with a “another backup is already in progress” message.
  • Blocks new agent runs via the shared refuseIfBackupInProgress guard wired into the assignments, peer-query, and webhook handlers. This closes the TOCTOU window between ensureAgentsIdle and docker pause.
Inspect or release the lock:
backup unlock is an emergency escape hatch. Only use it when you can confirm no backup is actually running (e.g. the previous CLI session crashed and the 1 h TTL has not yet fired). Forcibly releasing a live backup’s lock will let a second backup start alongside it, and the two will race on the docker pause/unpause sequence.

​
Examples

​
Nightly hot-spare backup with 14-day retention

A workspace on prod-server.example.com should produce a bundle every night, ship it to a separate backup host, and keep 14 days on disk locally as a fast-rollback safety net.
Paired with a crewship-backup.timer that fires at 02:37 daily (off-the-hour to avoid clustering with the rest of the fleet). The --use-keyring flag means the passphrase is read once and cached — the timer doesn’t have to redeliver it. The rotate step runs after the rsync so local-disk pressure is bounded even on long runs without a remote-side sweep.

​
Workspace migration to a new host

The acme workspace lives on crewship-old. You’re moving it to crewship-new to retire the old host. The destination already has its own workspaces, so a same-slug restore would conflict.
Once the new instance has parity (sanity-check via the UI), users get the new URL, and acme on the old host gets archived to cold storage before its workspace row is deleted.

​
DR drill with --dry-run

Every quarter the team proves the disaster-recovery bundle is actually restorable, without disturbing production state. Run the drill on a throwaway VM:
A backup.restore.dry_run audit entry shows up in the workspace journal — auditors looking at “did anyone restore prod?” can tell drill runs apart from real restores by the action name.

​
API reference

The backup surface is CLI-first — every operation is reachable via crewship backup … and every flag the CLI accepts maps to an HTTP body field. The full HTTP schema lives at /api-reference/backup; a quick orientation: All routes are mounted in internal/api/router_admin.go. The CLI talks to these directly when run against a remote host (--server flag) and falls back to in-process Go calls when run on the same machine as crewshipd — bypassing HTTP entirely for the host-shell use case. The two paths share the same handler functions, so flags work identically. Webhook payloads for backup.created / backup.restored / backup.failed events are documented separately under Webhooks; metric emissions are documented under Metrics. (Dry-run restores deliberately fire no webhook — they are not real restores.)

​
Streaming & memory bounds (large backups)

Restore and verify stream the sealed payload to a temp directory rather than buffering per-crew sections in a map[slug][]byte. Peak heap stays bounded by the zstd decoder window regardless of bundle size, so multi-GB restores run cleanly on small hosts. The extraction scratch directory is os.TempDir()/crewship-backup-… and is removed on Close(); a killed process leaves it behind for the next os.TempDir cleanup.

​
Passphrase keyring

The --use-keyring flag on create and restore caches the workspace passphrase in ~/.crewship/backup-keyring.enc so scripts and repeat rehearsals don’t re-prompt. The file is an AES-256-GCM-encrypted JSON map keyed by workspace ID, using the same v1:<base64> envelope as the credstore — without the host’s ENCRYPTION_KEY the contents are unreadable even if the file leaks.
Semantics worth knowing:
  • No silent fallback on failure. Opening the keyring or reading an entry reports the real error and aborts; only ErrKeyringEntryNotFound (first use on this workspace) falls through to a prompt.
  • Write failures are non-fatal on create. The bundle is already written when the keyring save runs; the CLI logs a warning and continues.
  • Keyring is local to the operator’s host. --use-keyring always writes to ~/.crewship/ on the invoking machine — even if a future remote bundle backend (S3 / GCS) is configured, the passphrase never travels with the bundle.
  • Single-process mutex, not file-locked. Two concurrent CLI invocations against the same workspace are last-write-wins (the file is small and the failure mode is “one passphrase lost, never data corruption”). Filesystem-level locking is on the v0.2 roadmap.
  • --passphrase-file takes precedence. When both flags are passed, the file wins and the keyring is not consulted.

​
Webhooks

Set CREWSHIP_BACKUP_WEBHOOK_URL (and CREWSHIP_BACKUP_WEBHOOK_SECRET) on the server process to receive a signed POST for each backup lifecycle event. Delivery is fire-and-forget from a goroutine — a slow or down webhook never blocks the backup run.
Each event is JSON with the shape:
Events: backup.created, backup.failed, backup.restored. Each request carries an X-Crewship-Signature: sha256=<hex> header — HMAC-SHA256 over the raw body using CREWSHIP_BACKUP_WEBHOOK_SECRET. Receivers must verify the signature (same scheme as Crewship’s inbound webhooks; validate via webhook.ValidateHMAC after stripping the sha256= prefix). The secret is required whenever URL is set — sending a body unsigned would let any network listener forge events to a downstream consumer that trusts the feed. URLs with userinfo or query strings are redacted before ever appearing in logs / audit rows, so basic-auth credentials or signed-URL tokens do not leak.

​
Metrics

GET /api/v1/admin/backups/metrics (instance OWNER only; see below) returns a point-in-time snapshot of process-lifetime counters. The numbers reset on restart — persistent observability belongs in the audit log and its dashboards.
Duration quantiles are approximated from an in-memory ring buffer — fine for the dozens-to-hundreds of samples a single host accumulates between restarts; not a general-purpose histogram. For long-horizon reporting, ingest the backup.* rows from audit_log.

​
Instance-scope backup

An instance-scope backup bundles every workspace on a Crewship host plus the cross-workspace surfaces that make the install usable — the credstore, the auth signing secret, and the instance identity (instance_config.hostname). It is the disaster-recovery path for an entire host, not a normal operational tool. Key differences from workspace/crew scope:
  • Access control. Gated by the CREWSHIP_OWNER_EMAIL env var (server-level OWNER), not workspace role. A workspace OWNER / ADMIN on their own is refused with HTTP 403.
  • Rate limit. One instance backup per user per sliding hour. A runaway cron cannot DoS the host.
  • Encryption is recipient-only. --passphrase-file is refused for this scope — the surface is too broad (every workspace’s secrets in one blob) to trust a brute-forceable passphrase. Callers must supply an AGE age1… X25519 public key and hold the matching private key offline.
  • Cross-host restores force session-key rotation. The bundle records the source hostname; a restore onto a different target invalidates every existing JWE session to prevent source-host tokens from remaining valid after DR.
Full threat model, crypto chain, and operational checklist: Security → Instance-Scope Backup Security.

​
Admin UI

A Backups tab lives in /admin for OWNER / ADMIN users. It wraps the same REST endpoints the CLI drives and adds:
  • A status banner for the advisory lock (who holds it, TTL remaining).
  • Create / restore dialogs with passphrase input (no keyring — the keyring is a CLI-side convenience; the browser never sees ~/.crewship/).
  • An inspect panel that renders the plaintext manifest without decrypting the payload (same as crewship backup inspect).
  • A bundle list with size, scope, format version, and created-at, fetched via hooks/use-backups.ts.
The UI does not expose instance-scope operations — they remain CLI + env-gated to reduce blast radius from a compromised admin session.

​
Known caveats (v0.2 roadmap)

The following items are intentionally deferred to v0.2. They do not block production use but are worth knowing.
  • preBackup / postBackup hooks — no user-defined hooks yet. If your workspace has services that need an app-level flush, run them manually before invoking crewship backup create.
  • Remote backends (S3 / B2 / GCS) — bundles live on the server’s local disk only. The storage layer is now abstracted behind a StorageOps interface so a future backend swap won’t require a second refactor of every call-site. Today: use scp / rclone / restic to ship bundles off-box, or stream a single bundle via GET /api/v1/admin/backups/download.
  • Scheduled backups — no built-in scheduler. Wrap crewship backup create in cron or systemd.timer.
  • Forward migration replay hooks. The plumbing is wired — migrations can register a per-version restoreBackfill function and the restorer walks the applied ∖ manifest set in ascending order after the main transaction commits — but no migration registers a hook yet. A failed backfill surfaces as ErrRestoreBackfillFailed; the restored rows are visible but may be missing backfilled columns until an admin investigates.
  • Cross-process keyring lock. The per-process mutex does not cover two concurrent CLI invocations racing on the same keyring file.

​
Common pitfalls

Losing the passphrase = losing the data. AGE bundles are not recoverable without the passphrase (or X25519 private key). There is no master key, no support escape hatch. Store the passphrase in a separate trust zone from the bundle — a password manager on a different host, a sealed envelope, anything that doesn’t share a failure mode with the disk holding the .tar.zst.
The advisory lock exists precisely because two concurrent backups race on the docker pause / docker unpause sequence. Only ever clear a lock when you can prove its holder is dead (CLI session crashed, host rebooted) and the 1-hour TTL hasn’t fired yet. When in doubt, wait for the TTL.
Like the Admin CLI trap — if the server runs with a custom data dir but you invoke crewship backup … without the same env var, the CLI defaults to ~/.crewship and operates on an empty/separate database. Backups succeed but capture nothing useful; restores produce “workspace doesn’t exist” errors. Export the same CREWSHIP_DATA_DIR the server uses (there is no --data-dir flag — the env var is the only override).
Restore takes the advisory lock but a running crewshipd may still hold open handles on tables the restorer wants to truncate. Symptoms range from “database is locked” SQLite errors to a half-applied restore that leaves the workspace in a non-bootable state. The lock guards against concurrent backups, not against the server’s own writes — stop the service first.
A bundle written by format v5 will restore on v5, v6, and v7 servers; v8 onward, the compatibility window has rolled past it. If you’re restoring from cold storage that’s been sitting for a year+, verify the manifest’s format_version against the destination first with crewship backup inspect.
Passing two raises a CLI error rather than silently picking one. A bundle encrypted with --recipient age1… will not decrypt with a passphrase, and vice versa — match the restore flag to the original create flag.
crewship backup restore will not overwrite an existing workspace or crew under the same slug. If the existing one is stale and you want to replace it, delete it first (a crew via crewship crew delete; a workspace via crewship workspace delete <slug> --confirm <slug> — OWNER only, refused if it is your only workspace, see the workspace CLI); if you want both side-by-side, use --as-workspace=<new-slug> and then crewship crew start.
Peak RAM is fixed by the zstd window, but the extraction scratch directory at $TMPDIR/crewship-backup-* needs ~2× the bundle size on disk. Restoring a 4 GB bundle onto a host with 6 GB free in /tmp will fail mid-stream. Mount /tmp on the data volume or set TMPDIR to a larger filesystem.
A non-TTY second invocation will block waiting for stdin. Either pass --passphrase-file, pipe the passphrase via stdin redirect, or use --use-keyring so the first call seeds the cache.
User code in /workspace, the crew and per-agent memory trees under /crew, anything mounted into the agent container — all included in bundles at format_version 3 and above (see the memory warning under Bundle layout for what v1 and v2 bundles actually contain). Workspace-scope bundles also carry the content-addressed memory-version blob store ({MemoryRoot}/versions) behind memory_versions rows — see Memory-version blobs. AGE encryption is the only defence; treat a .tar.zst like any other backup of source code and credentials.
This is load-bearing — agents pull credentials from /secrets at runtime via Keeper, and including them in a portable bundle would defeat the SECRET-tier guarantee. Don’t add code that bypasses the secret-skipping filter without an equally rigorous out-of-band channel.

​
Security notes

  • The /secrets mount is never included in a bundle.
  • The workspace bind mount (user code), the per-crew and crew-shared memory trees under /crew (v3 bundles and above), and the workspace-tier memory-version blob store (memory_versions content, workspace scope only) are all included. AGE encryption of the payload is the only defence against leakage from physical bundle distribution — treat bundles like any other backup of your source tree.
  • Every backup.* event writes to audit_log with user, role, scope, sealed-payload SHA-256, and size. Dry-run restores write backup.restore.dry_run so auditors can tell rehearsals from real cutover.

​
Automatic pre-migration snapshots

Distinct from the manual crewship backup bundle system above: Crewship auto-snapshots the SQLite DB before any pending migration runs. This is the “binary upgrade went sideways, give me my data back” safety net.

​
How it works

Every crewship start calls database.SnapshotBeforeMigrate before database.Migrate. If any migrations are pending (rows in the migrations[] slice with version
MAX(version) FROM _migrations), the function:
  1. Resolves the SQLite file path from the connection DSN.
  2. Computes a target name: <dbpath>.pre-migrate-v<from>-to-v<to>-<UTC-RFC3339>.bak.
  3. Issues VACUUM INTO '<target>' — SQLite’s hot-copy mechanism. Safer than a plain file copy because it serializes against concurrent writers and produces a defragmented, WAL-checkpointed snapshot.
  4. Chmods the snapshot to 0600.
  5. Prunes older snapshots, keeping the 10 most recent per database file.
On error the boot aborts before any migration runs — silently continuing without a rollback point would defeat the entire purpose.

​
Opting out

Skips the snapshot. Useful for CI environments where the DB is ephemeral and snapshot I/O is just overhead.

​
Recovering from a bad migration

If crewship start succeeds, applies a migration, then exhibits runtime errors that point at schema drift:
1

Find the most recent pre-migration snapshot

2

Stop crewship

3

Replace the current DB with the snapshot

4

Run with the PREVIOUS binary (the one that wrote the snapshot)

The snapshot is a complete SQLite file — no special tooling needed. Tools like sqlite3 open it directly for inspection.

​
Limits of pre-migration snapshots vs manual backups

Use both. Pre-migration snapshots cover binary upgrades; crewship backup covers DR, host migration, legal hold, and forensic preservation. They are not interchangeable.
  • Backup API reference — REST endpoint shapes for every crewship backup subcommand.
  • Admin CLI — the host-shell write surface that backups protect against; always crewship backup create before running admin writes in prod.
  • Migrations Catalog — what migration ran when, and which restoreBackfill hook (if any) will fire on a cross-version restore.
  • Troubleshooting — recovering from a stuck migration or corrupt DB.
  • Security → Audit log — where backup.created, backup.restored, and backup.restore.dry_run events land.