Skip to main content

​
Backup API

REST endpoints for creating, listing, inspecting, verifying, restoring, rotating, and deleting backup bundles. All routes live under /api/v1/admin/backups and act on the workspace the request names. They admit an OWNER or ADMIN of that workspace, or an instance administrator, who need not be a member of it (the admin floor: RequireWorkspaceOrInstanceAdmin); handlers re-check with canAdministerInstance. Every mutation writes one row to audit_log. Routes are registered in internal/api/router_admin.go (domain split in May 2026); handlers live in internal/api/backup.go.
Every endpoint except metrics requires authentication and a workspace — workspace_id in the query or X-Workspace-ID, an id or a slug — and OWNER or ADMIN on it, or an instance administrator. Without a workspace an instance administrator gets 400. The metrics endpoint needs no workspace and admits only instance administrators — see its authorisation warning.

​
Authentication and response contract

Every request uses the workspace it names in the query or header; no endpoint accepts a workspace ID in its body. Requests without a body are marked explicitly below, and path-based operations take path as a query parameter. Statuses: successful JSON operations return 200 or 201, successful deletion returns 204, and invalid input, authentication, authorization, conflicts, and server failures use the status codes listed in each operation’s error table.

​
Endpoints

Path-based endpoints (inspect, verify, download, restore, delete) validate that the supplied path lives under the server’s default backups directory ($CREWSHIP_DATA_DIR/backups, or ~/.crewship/backups if unset) and belongs to the caller’s workspace. Symlinks are explicitly rejected during path validation — no escape via ln -s to read files outside the backup directory. Paths from other workspaces return 404 Not Found rather than 403, so the endpoint does not confirm existence of bundles outside the caller’s scope.

​
Create and list

Create new bundles and enumerate the ones already on disk for the current workspace.

​
POST /api/v1/admin/backups

Create a new backup bundle. Runs inline (seconds to minutes); no async job queue.

​
Request

​
Response — 201 Created

scope_level is omitted when the bundle has no explicit level recorded in its manifest. missing_container_crews names provisioned crews whose container was absent from the daemon — those crews are backed up DB-rows-only, so the export is incomplete rather than silently whole. The key is always present in this response: for a complete bundle it is null (not []), meaning every provisioned crew’s filesystem was collected. In the bundle’s own manifest the field is omitted when empty.
contents.crews[].memory_included means “the crew and per-agent memory trees under /crew are in this bundle” only from format_version 3. At v1 and v2 the flag was set unconditionally for any crew that had a container, and the section behind it held /output, not memory. Read it together with format_version, or use the CLI, which already does. From v3 every per-crew inclusion flag is derived from what the collector observed writing: memory_included (files inside a .memory directory), crew_files_included (any file in the /crew section, memory or not), output_included (/output), workspace_included, volumes_included, system_included.memory_included counts FILES, not tar entries, and only those inside a .memory directory. The runtime creates crews/<id>/shared and crews/<id>/agents at container creation, so an entry count would report memory for every provisioned crew regardless of whether an agent ever wrote one.
New bundles are written at format_version 3 (backup.FormatVersion, bumped v2→v3 on 2026-08-03 when the payload gained the crew/<slug> section carrying the crew and per-agent memory trees). Restore enforces an N-2 compatibility window: this binary reads format_version in [MinSupportedFormatVersion, FormatVersion] = [1, 3]. A bundle newer than the reader returns ErrFormatTooNew (400); older than the window returns ErrFormatTooOld (400), whose message names the exact crewship backup convert command that writes a readable copy (see Compatibility).

​
Page projects

A workspace-scope bundle whose page_project_* rows reference source projects, compiled artifacts or Git checkpoints carries them in a page-projects/ section of the sealed payload — see the bundle layout. It is collected under the page project store’s shared lease, so the rows and the files agree; a referenced source or artifact missing on disk, or an instance without CREWSHIP_PAGE_PROJECTS_PATH while the rows reference one, fails the create with 500 rather than writing a bundle that cannot be restored. The section is absent, not empty, when nothing references it.

​
Errors

Audit action: backup.create.

​
GET /api/v1/admin/backups

List bundles from the default backups directory ($CREWSHIP_DATA_DIR/backups, or ~/.crewship/backups if unset) filtered to the caller’s workspace (bundles of other workspaces are hidden from the response).

​
Response — 200 OK

Entries are sourced from backup_catalog first (reconciled against disk on each call); a missing/empty catalog falls back to a filesystem scan. scope_level is omitted when the source manifest did not record it.

​
Locking and metrics

Inspect or force-release the per-workspace advisory lock, and read process-lifetime backup counters.

​
GET /api/v1/admin/backups/status

Report the workspace’s advisory lock.

​
Response — 200 OK

held: false omits the detail fields. The lock TTL is 1 hour (backup.DefaultLockTTL). Missing workspace context returns 400 (not 401).

​
DELETE /api/v1/admin/backups/status

Force-release the per-workspace lock regardless of owner. Intended as an emergency escape hatch when a crashed backup left a stale lock behind. Returns 204 No Content. Audit action: backup.unlock.

​
GET /api/v1/admin/backups/metrics

Return a point-in-time snapshot of process-lifetime counters for the backup subsystem — created / failed / restored totals, size bytes, approximate duration quantiles, and per-workspace lock-hold seconds. Counters reset on every restart; persistent reporting should read audit_log instead.
Authorisation is stricter than the other backup endpoints. The counters are instance-wide, so this route admits only an instance administrator — the CREWSHIP_OWNER_EMAIL owner or a named instance admin — and needs no workspace. A workspace OWNER / ADMIN on their own is refused with 403 Forbidden ({"error": "instance administrator required"}).

​
Response — 200 OK

Duration quantiles are approximated from an in-memory ring buffer (tens-to-hundreds of samples between restarts) — adequate for a dashboard tile, not a general-purpose histogram.

​
Errors

​
Inspect, verify, and self-test

Read a bundle’s manifest, recompute its checksum, or run an end-to-end canary without leaving a bundle on disk.

​
POST /api/v1/admin/backups/self-test

Per-crew round-trip canary: collect → destroy a canary marker inside the crew container → restore → verify → cleanup. Used by the seed CLI and CI harness to validate the backup pipeline end-to-end without leaving a bundle on disk. Lightweight: no encryption, no DB dump. Gated by workspace OWNER/ADMIN (canRole(role, "manage")), the same as the other backup admin endpoints — not by CREWSHIP_OWNER_EMAIL. Request body:
Response: 200 OK — backup.SelfTestResult JSON (collect/restore/verify status and timing for each phase).

​
GET /api/v1/admin/backups/inspect

Return the plaintext MANIFEST for a bundle without decrypting the payload.

​
Query

The request has no body. Response: 200 OK with the manifest JSON.

​
Response — 200 OK

Returns the manifest JSON as produced by backup.Inspect (internal/backup/manifest.go).

​
GET /api/v1/admin/backups/verify

Recompute the sealed-payload SHA-256 and compare against the MANIFEST — integrity. Never decrypts an ENCRYPTED bundle, so this needs no passphrase and is safe to run unattended against a whole directory of nightly bundles. For an UNENCRYPTED bundle, also compares the payload’s actual per-table row counts against manifest.contents.table_row_counts (#2009) — completeness. This is what closes the gap that let a bundle which dumped zero rows for a table (a create-side scoping bug, or corruption) still report valid: true: that divergence now sets valid: false too, not only a checksum mismatch.

​
Query

The request has no body. Response: 200 OK with the verification object.

​
Response — 200 OK

valid: false sets error to a human-readable reason (checksum mismatch, truncated payload, unreadable bundle, a payload/manifest row-count divergence, …). completeness_checked reports whether the row-count comparison ran at all — false means not evaluated, never “confirmed complete” and never “confirmed incomplete”. It is false for an encrypted bundle (completeness_skip_reason explains and points at restore --dry-run, which decrypts and makes the same comparison — see below) and for a bundle written before #2009, which carries no recorded counts to compare against. table_row_count_mismatches is non-empty only when completeness_checked is true and a divergence was found; each entry is {table, recorded, actual}. scope_shortfalls repeats the manifest’s create-time contents.scope_reconciliation.shortfalls: each entry is {table, reachable, missing}, the workspace rows the schema tied to the workspace that the dump did not export. It is checked for encrypted bundles too, and a non-empty list makes valid false.

​
Retention, transfer, and restore

Apply retention policy, stream bundles off-box, restore into the current workspace, or delete bundles from disk.

​
POST /api/v1/admin/backups/rotate

Apply retention policy to the caller’s workspace. Per-workspace only — another workspace’s bundles are never enumerated.

​
Request

At least one of keep_last / keep_days must be positive (0 disables the rule). Negatives are rejected. keep_last is a floor: the newest keep_last bundles are always kept, however old. keep_days applies only beyond it (alone, the floor is one bundle). Bundles pinned through POST /api/v1/admin/instance/backups/bundles/pin are never deleted. Crew bundles are counted separately from workspace bundles.

​
Response — 200 OK

With dry_run: true the paths are returned but the files stay on disk and no audit rows are written. Audit action (per deleted bundle, non-dry-run only): backup.rotate.

​
GET /api/v1/admin/backups/download

Stream the raw bundle bytes over HTTP for scp-free transfer off-box. The request has no body. Response: 200 OK with an application/zstd attachment; 400 is returned for a missing or unsafe path, 404 for a bundle outside the workspace or no longer on disk, and 500 for a file error. Response headers:
  • Content-Type: application/zstd
  • Content-Disposition: attachment; filename="<file>"
  • Cache-Control: no-store
  • X-Content-Type-Options: nosniff
Audit action: backup.download.

​
POST /api/v1/admin/backups/restore

Restore a bundle into the current workspace.

​
Request

​
Response — 200 OK

capability_tokens_reminted is non-null only on a forked restore (as_workspace / as_crew): per-table counts of the capability tokens that were re-minted — invitations, port exposures, pipeline and page webhooks, public links — so no secret that worked against the source works against the fork. null on a plain restore, which re-keys nothing. docker_phase_skipped: true when as_workspace / as_crew rewrote the slug — container state is not landed. dropped_crew_filesystems names the crews that lost it. Finish the restore by starting each of them (POST /api/v1/crews/{crewId}/container-start) and then calling this endpoint again with files_only: true and the new workspace in context. It must be container-start and not /provision: files_only writes each section by exec’ing into the crew’s container, so the container has to be running, and provision only builds the image. Re-calling WITHOUT files_only after a rename does not work and is not the recovery: the forked workspace matches neither the bundle’s id nor its slug, so the tenant guard returns 403 — and its rows are already present under new ids anyway. crews_count is what the bundle describes; crews_restored is how many crews actually received container state. The two differ whenever a crew carries no filesystem sections or has no provisioned container on this instance. A files_only call that would land nothing is rejected rather than returning 200 with the bundle’s count — an operation reporting success without having written anything is the failure mode this whole area exists to have stopped. security_level_clamped counts the restored credentials whose security_level was not one of the defined tiers (L1–L4) and were therefore written at the strictest tier instead — every other writer of that column validates it, so restore does too. security_level_clamps is a bounded sample naming the affected credentials and the value the bundle carried; the full list of what happened is also written to the journal as credential.security_level.clamped. On a dry_run both fields report what would be clamped. Re-set each affected credential with PATCH /api/v1/credentials/{id} (security_level) or crewship credential update <name> --security-level N. columns_dropped counts the values this restore discarded because the bundle named a column this instance’s schema does not have; dropped_columns names them as {table, column, rows}, bounded to 20 pairs while the count stays exact. On a dry_run both report what would be discarded, which is the point at which you can still do something about it. Dropping unknown columns is deliberate — it is what lets a bundle from a newer Crewship restore into an older instance — but it is not always harmless, and rows_inserted cannot show the difference. A row whose remaining columns satisfy the table lands without those values. A row that needed a dropped column to satisfy a NOT NULL or a primary key does not land at all, and INSERT OR IGNORE reports neither the violation nor the loss. That is what a non-zero columns_dropped is warning about: the bundle was written against a different schema, and this restore is incomplete in a way nothing else in the response records. Check each table named before treating the restore as done. Restoring a bundle taken before a non-additive migration (one that re-keys or renames rather than adds) is the case that produces it. issue_counters_migrated (#2034) is the one exception the restore repairs instead of merely reporting: a bundle taken before migration 20260820125000_issue_counters_prefix_scope (#1797) carries issue_counters rows keyed on crew_id, which the current table does not have a column for — without this, the row would land in columns_dropped and, because crew_id was part of the old primary key, never actually restore. Each row is resolved instead, by crew, to its workspace and effective prefix (from the bundle’s own crews table, or the target’s if the crew already exists there) and written under the current (workspace_id, prefix) key; two crews sharing an effective prefix collapse onto one row, keeping the higher next_number. A row whose crew cannot be resolved anywhere still falls through to columns_dropped — it is not guessed at. On a dry_run, issue_counters_migrated reports what would be migrated. payload_row_count_mismatches and rows_inserted_shortfalls (#2009) are two distinct completeness comparisons, both [{table, recorded, actual}]:
  • payload_row_count_mismatches is the same comparison GET /api/v1/admin/backups/verify makes for an unencrypted bundle, run here regardless of encryption — restore already has the decrypted payload for other reasons. A non-empty result means the bundle’s decrypted contents disagree with what its own manifest recorded; the bundle is not what it claims to be, independent of whatever the target does with it.
  • rows_inserted_shortfalls compares what actually landed (RowsInsertedByTable) against the manifest’s recorded counts, per table. A table can come up short here even when payload_row_count_mismatches is empty — a dropped key column (columns_dropped, above) or a primary-key collision INSERT OR IGNORE swallowed are the usual causes, and each of those already has its own more specific report; this is the summary that catches whatever isn’t covered by a more specific one. issue_counters is discounted for the rows a pre-#1797 migration legitimately collapsed (issue_counters_migrated, above) before this comparison runs, so a genuine shortfall on that table still surfaces. Empty on a dry_run — nothing was inserted to compare.
journal_entries_resigned and journal_checkpoints_resigned (#2226) are non-zero only on a forked restore (as_workspace / as_crew). A fork regenerates the ids the journal hash chain commits to, so the chain is re-signed under a new genesis and no longer links back to the source’s history; the two counts say how many entries and compaction checkpoints were re-signed, and the same fact is written into the fork’s own journal as a backup.chain_resigned entry. Both are 0 on a plain restore, which remaps nothing — a clean verify on the fork is integrity of the fork’s chain, not provenance from the source. See Backup → Forked restores. Neither field causes the restore to fail or roll back; both are reported so the operator can decide, the same way columns_dropped is.

​
Errors

Audit action: backup.restore (or backup.restore.dry_run for dry_run: true).

​
DELETE /api/v1/admin/backups

Delete a bundle from disk. Returns 204 No Content. The request has no body. Response: 204 No Content; 400 indicates a missing or unsafe path, 404 indicates a bundle outside the workspace, and 500 indicates a delete failure.

​
Query

Audit action: backup.delete.

​
Error mapping

internal/api/backup.go:statusForBackupError maps sentinel errors to HTTP status using errors.Is, so the backup package can rework error wording without breaking HTTP contracts:

​
See also