Credential Encryption

All credentials in Crewship are encrypted at rest using AES-256-GCM. The implementation in internal/encryption/encryption.go is designed for cross-compatibility between Go and TypeScript.

Encryption Scheme

Parameter	Value
Algorithm	AES-256-GCM
Key size	256 bits (32 bytes, hex-encoded = 64 chars)
IV (nonce) size	16 bytes (not the standard 12 bytes)
Auth tag size	16 bytes
Key source	`ENCRYPTION_KEY` environment variable
Key format	Hex-encoded string

The IV/nonce is 16 bytes, not the standard 12 bytes. This is set via cipher.NewGCMWithNonceSize(block, 16) for Go/TypeScript compatibility. Changing this breaks all stored credentials.

Wire Format

Encrypted values are stored as versioned base64 strings:

v1:base64(IV || AuthTag || Ciphertext)

Byte Layout

+--------+----------+------------+
| IV     | AuthTag  | Ciphertext |
| 16 B   | 16 B     | variable   |
+--------+----------+------------+
  bytes    bytes      bytes
  0-15     16-31      32+

This is a custom byte order (IV || AuthTag || Ciphertext), not the standard Go GCM output format (Ciphertext || AuthTag). The reordering is necessary for compatibility with the TypeScript implementation in lib/encryption.ts.

The byte layout is IV(16) || AuthTag(16) || Ciphertext. This differs from Go’s default GCM Seal output which produces Ciphertext || AuthTag. The encryption code explicitly splits and reorders these components. Changing this layout breaks all stored credentials and cross-language compatibility.

Encryption Flow

// internal/encryption/encryption.go - Encrypt function

Load key from ENCRYPTION_KEY env var (hex-decoded to 32 bytes)
Create AES cipher block
Create GCM with 16-byte nonce: cipher.NewGCMWithNonceSize(block, 16)
Generate 16 random IV bytes from crypto/rand
Seal: sealed = gcm.Seal(nil, iv, plaintext, nil)
   // Go GCM output: ciphertext + authTag (last 16 bytes)
Split sealed into ciphertext and authTag
Reorder: combined = IV + authTag + ciphertext
Encode: "v1:" + base64.StdEncoding.EncodeToString(combined)

Decryption Flow

// internal/encryption/encryption.go - Decrypt function

Parse version prefix: "v1:base64data" or legacy "base64data"
Look up key for version (ENCRYPTION_KEY or ENCRYPTION_KEY_V2, etc.)
Base64 decode (try standard first, fall back to raw/no-padding)
Validate minimum length (32 bytes: 16 IV + 16 AuthTag)
Extract: iv = data[0:16], authTag = data[16:32], ciphertext = data[32:]
Reconstruct Go GCM format: sealed = ciphertext + authTag
Decrypt: gcm.Open(nil, iv, sealed, nil)

Key Versioning

Envelopes are prefixed with a version identifier (e.g., v1:) to support key rotation. The version NEW envelopes are minted with defaults to v1 and is switched with the CREWSHIP_ENCRYPTION_KEY_VERSION environment variable during a rotation. Decryption always honours the per-envelope prefix, so any generation whose key is still in the environment remains readable.

Key Resolution

Version	Environment Variable
`v1` (default)	`ENCRYPTION_KEY`
`v2`	`ENCRYPTION_KEY_V2`
`vN`	`ENCRYPTION_KEY_VN`

On decrypt, if the version-specific env var is not set, resolution falls back to ENCRYPTION_KEY (legacy compatibility). On encrypt, the resolution is strict: minting vN envelopes requires ENCRYPTION_KEY_VN to be present — a silent fallback would stamp envelopes with a version whose key they were not encrypted with. crewship start validates the active version’s key at boot and refuses to start on a misconfigured rotation.

Key Source Visibility

On first boot crewship start auto-generates ENCRYPTION_KEY and persists it to <dataDir>/secrets.env (mode 0600) — which on default installs is the same volume as the SQLite database. A copied disk or volume backup therefore carries both the ciphertext and the key. Crewship surfaces this:

A startup WARN log fires on every boot while the key is auto-generated/colocated.
GET /api/v1/admin/health (CLI: crewship system health) reports encryption_key_source: "external" (operator-injected env var), "generated" (auto-bootstrapped, colocated), or "unknown".

For stronger at-rest protection, supply ENCRYPTION_KEY from an external secret store (systemd EnvironmentFile=, Kubernetes secret, Vault) and remove the entry from secrets.env.

No key configured (fail-closed)

Secret writes fail closed. If no usable encryption key resolves for the active key version, Crewship refuses to store the secret rather than silently writing it to the database in plaintext. This closes a compatibility shim that shipped with the webhook-secret work (#1072/#1029) and was never given a sunset: before this change, a deployment with no ENCRYPTION_KEY stored webhook signing secrets as plaintext and the only trace was a log line nobody was reading.

What an operator sees

The write fails and the API returns 500. For the webhook-secret endpoints (agent webhook-secret rotate, pipeline-webhook create) the response body itself is actionable — it names ENCRYPTION_KEY and the opt-out flag — and the server log carries the full reason, naming the env var that is missing or malformed:

refusing to store a secret in plaintext: no usable encryption key is configured
(ENCRYPTION_KEY is not set (required to mint v1 envelopes)). Generate one with
`openssl rand -hex 32` and set ENCRYPTION_KEY, or set
CREWSHIP_ALLOW_PLAINTEXT_SECRETS=true to accept plaintext storage

How to fix it

Generate a key and put it in the environment, then restart:

openssl rand -hex 32     # 32 bytes → 64 hex chars
export ENCRYPTION_KEY=<the value>

On a normal install you should never hit this: crewship start bootstraps ENCRYPTION_KEY on first boot (persisting it to <dataDir>/secrets.env) and then calls VerifyCurrentKey() — it refuses to start if the active key version’s env var is missing or malformed. So a key-less deployment is told at boot, not at first secret write. The fail-closed encrypt path is the backstop for embedded/alternate entry points that skip that bootstrap, and for a mid-rotation CREWSHIP_ENCRYPTION_KEY_VERSION pointing at a key that has not been supplied yet.

Escape hatch: CREWSHIP_ALLOW_PLAINTEXT_SECRETS. Setting it to true (also accepted: 1, yes, on — anything else means false) restores the old behaviour: with no key, secrets are written to the database in plaintext.Every such write emits a WARN, deliberately not deduplicated, so the count of unprotected secrets is auditable:

level=WARN msg="storing secret UNENCRYPTED at rest — no encryption key configured
and CREWSHIP_ALLOW_PLAINTEXT_SECRETS=true is set; set ENCRYPTION_KEY
(openssl rand -hex 32) and re-run `crewship admin reencrypt` to protect existing rows"
opt_out_env=CREWSHIP_ALLOW_PLAINTEXT_SECRETS key_env=ENCRYPTION_KEY

The secret value itself is never logged. Treat this flag as a short-lived transition aid, not a configuration: anything written while it is on stays plaintext in the database until you configure a key and run crewship admin reencrypt.

Reading already-stored plaintext

Fail-closed applies to writes only. Values already stored in plaintext by an older build (or under the opt-out) continue to be read back unchanged, with or without a key present and regardless of the flag — the read path discriminates on the vN: envelope prefix. Turning fail-closed on therefore cannot brick an existing install’s stored secrets.

The migration backfill waits for a key

Upgrading to the release that introduced webhook-secret encryption runs a one-time migration (v140) that encrypts any webhook/signing secrets previously stored in plaintext. If no usable ENCRYPTION_KEY is configured when that migration runs, it skips instead of failing — a migration must never break boot — and logs a WARN that the secrets are left plaintext at rest. The backfill is effectively deferred: the migration does not run again later, so after configuring a key, run:

crewship admin reencrypt

to perform the backfill the skip deferred. Newly written secrets are protected by the fail-closed write path either way.

Rotating the Master Key

crewship admin reencrypt re-encrypts every stored envelope to the current key version, so the old key can actually be retired instead of living in the environment forever. Runbook (rotating from v1 to v2):

# 1. Generate the new key
openssl rand -hex 32

# 2. On the server, add the new key AND flip the active version,
#    keeping the old key available for decryption:
export ENCRYPTION_KEY=<old key>                # unchanged
export ENCRYPTION_KEY_V2=<new key>
export CREWSHIP_ENCRYPTION_KEY_VERSION=v2
#    …then restart crewship. New writes now mint "v2:" envelopes.

# 3. Re-encrypt everything that is still on the old key:
crewship admin reencrypt
# Re-encrypted: N   Skipped: M   Failed: 0

# 4. Once the run reports failed: 0, no row references the old key.
#    Retire it: keep ENCRYPTION_KEY_V2 + CREWSHIP_ENCRYPTION_KEY_VERSION=v2,
#    and remove/replace the old ENCRYPTION_KEY value.

The operation is idempotent — envelopes already at the current version are skipped — so an interrupted run can simply be repeated. Undecryptable values (no configured key opens them) are counted under failed and left untouched; the CLI exits non-zero in that case so a scripted rotation never retires the old key on a false success. The walk covers every envelope-bearing column: credentials.encrypted_value, credentials.encrypted_refresh_token, credentials.oauth_client_secret_enc, credentials.oauth_refresh_token_enc, credential_rotations.old_value, notification_channels.secret_enc, composio_settings.encrypted_api_key, oauth_states.code_verifier, escalations.resolution (credential-type escalations only), and the two webhook-secret columns agents.webhook_secret and pipeline_webhooks.signing_secret.

Webhook secrets are encrypted at rest, fail-closed. agents.webhook_secret and pipeline_webhooks.signing_secret are encrypted whenever a usable key resolves; with no key the write is rejected rather than downgraded to plaintext (see No key configured below). Rows written in plaintext by older builds keep working — they are read back as-is. During master-key rotation such a bare (non-enveloped) webhook value is reported as skipped, not failed, so the failed=0 ⇒ retire the old key criterion stays honest. Migration v140 backfills existing plaintext webhook secrets to envelopes when a key is present.

One known edge case: databases created in the brief pre-release window before encrypt-on-resolve landed (March 2026) may hold credential-type escalation resolutions stored as plaintext. Those rows always count under failed (they are left untouched — no corruption), so a failed: 0 state is unreachable on such a database without clearing the affected historical escalations.resolution rows manually.

Backup bundles created before the rotation contain credential envelopes under the old key. Restoring one requires the old key to still be resolvable (keep it archived offline), or take a fresh backup after the re-encryption run.The CLI-side backup keyring (~/.crewship/backup-keyring.enc) is a separate rotation domain: it is encrypted with the CLI host’s own ENCRYPTION_KEY, not the server’s, so the server-side run never touches it. If you rotate the CLI host’s key, re-store the affected passphrases (crewship backup create --use-keyring re-writes the entry) or keep that host’s old key resolvable.

Legacy Format Support

The decryptor accepts both:

Versioned: v1:base64data (current format)
Legacy: base64data (no version prefix, treated as v1)

Base64 decoding tries standard encoding first, then falls back to raw (no padding) for TypeScript compatibility.

Cross-Language Compatibility

The encryption format is shared between:

Go: internal/encryption/encryption.go
TypeScript: lib/encryption.ts

Both implementations must agree on:

IV size: 16 bytes
Byte order: IV || AuthTag || Ciphertext
Base64 encoding: standard or raw
Key derivation: direct hex decode (no KDF)

Security Properties

Property	Implementation
Confidentiality	AES-256-GCM encryption
Integrity	GCM authentication tag
Freshness	Random 16-byte IV per encryption
Key isolation	Key only in environment variable, never in DB
Minimum ciphertext	32 bytes (IV + AuthTag with empty plaintext)

Generating an Encryption Key

# Generate a 256-bit key (32 bytes, 64 hex characters)
openssl rand -hex 32

# Example output:
# a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2

The encryption key must be exactly 64 hex characters (32 bytes). A shorter or longer key will cause hex.DecodeString to fail or produce the wrong key length for AES-256.

Credential Lifecycle

User enters credential value
    |
    v
API handler calls encryption.Encrypt(plaintext)
    |
    v
"v1:base64(IV||AuthTag||Ciphertext)" stored in DB
    |
    v (on agent start)
Orchestrator calls encryption.Decrypt(ciphertext)
    |
    v
Plaintext piped to sidecar via stdin JSON
    |
    v
CredStore (in-memory, never on disk)
    |
    v (on HTTP request)
Sidecar injects into outbound request headers

Credentials are transmitted to the sidecar via stdin JSON — not environment variables. This prevents credential leakage through /proc/environ or process listing tools.

Get Started

Guides

Security

Configuration

Manifest Reference

Credential Encryption

Credential Encryption

Encryption Scheme

Wire Format

Byte Layout

Encryption Flow

Decryption Flow

Key Versioning

Key Resolution

Key Source Visibility

No key configured (fail-closed)

What an operator sees

How to fix it

Reading already-stored plaintext

The migration backfill waits for a key

Rotating the Master Key

Legacy Format Support

Cross-Language Compatibility

Security Properties

Generating an Encryption Key

Credential Lifecycle

​Credential Encryption

​Encryption Scheme

​Wire Format

​Byte Layout

​Encryption Flow

​Decryption Flow

​Key Versioning

​Key Resolution

​Key Source Visibility

​No key configured (fail-closed)

​What an operator sees

​How to fix it

​Reading already-stored plaintext

​The migration backfill waits for a key

​Rotating the Master Key

​Legacy Format Support

​Cross-Language Compatibility

​Security Properties

​Generating an Encryption Key

​Credential Lifecycle

Credential Encryption

Encryption Scheme

Wire Format

Byte Layout

Encryption Flow

Decryption Flow

Key Versioning

Key Resolution

Key Source Visibility

No key configured (fail-closed)

What an operator sees

How to fix it

Reading already-stored plaintext

The migration backfill waits for a key

Rotating the Master Key

Legacy Format Support

Cross-Language Compatibility

Security Properties

Generating an Encryption Key

Credential Lifecycle