Audit Log

Crewship maintains an append-only audit log that records every mutation in the system. The audit log provides a complete trail for security review, compliance, and debugging.

What Is Audited

The audit_logs table currently captures these mutations. Other entities (crews, missions, credentials) emit journal events but do not yet have WriteAuditLog calls wired into their handlers:

Category	Actions	Source
Agents	`create`, `update`, `delete`	`internal/api/agents_create.go`, `agents_update.go`, `agents_query.go`
Backups	`backup.create`, `backup.restore`, `backup.unlock`, `backup.rotate`, `backup.delete`, `backup.download`	`internal/api/backup.go`, `backup_admin.go`

Keeper security decisions are tracked separately in the keeper_requests table (see below), not in audit_logs.

Keeper Decision Auditing

Keeper security decisions are stored in the keeper_requests table with detailed fields:

Field	Description
`intent`	The agent’s stated reason for access
`request_type`	`access` (credential read via `/keeper/request`) or `execute` (command run via `/keeper/execute`)
`command`	Shell command (for execute requests)
`decision`	`ALLOW`, `DENY`, or `ESCALATE`
`reason`	LLM-generated explanation
`risk_score`	Numeric risk assessment (1-10)
`ollama_prompt`	Full prompt sent to the Keeper LLM
`ollama_raw_response`	Raw LLM response for review

The decision history is append-only

keeper_requests holds the current state of a request. It is written PENDING and then updated in place once the gatekeeper decides — so on its own it cannot tell you that a request was ever pending, how long it sat there, or whether a decision was later rewritten. Every state transition is therefore also appended to keeper_request_events:

Column	Description
`request_id`	The `keeper_requests` row this transition belongs to
`seq`	1-based, monotonic per request. A gap or duplicate is itself a signal
`state`	The state entered: `PENDING`, `ALLOW`, `DENY`, `ESCALATE`, `DUPLICATE_SUPPRESSED`, or a Phase-2 verdict
`actor_type` / `actor_id`	Who caused it — `agent` (raised the request), `keeper` (decided it), `user` (resolved an escalation), `system` (dedup suppression)
`reason`, `risk_score`, `exit_code`	The decision detail at that transition. `NULL` where not yet applicable — a `PENDING` has no risk score, and no exit code exists before the command ran
`recorded_at`	When the transition happened

The agent, crew, credential, intent and command are copied onto every transition rather than joined, so the record stays readable and self-describing if the operational keeper_requests row is ever pruned.

keeper_request_events is append-only at the database level: a BEFORE UPDATE trigger aborts any attempt to rewrite a recorded transition. A decision cannot be edited, only superseded by a further transition — which appears as an extra row rather than a silently changed value. Immutability that depended on every future caller remembering to insert instead of update would not be immutability.DELETE is deliberately not blocked: workspace_id carries ON DELETE CASCADE so workspace teardown and backup --replace still work. Deletion is covered by the other half of the model — every keeper decision is also mirrored into the hash-chained journal, where a removal shows up as a sequence gap that only a signed checkpoint can legitimately bridge.

Read the history with:

crewship keeper requests                  # current state of recent requests
crewship keeper history <request-id>      # every transition, oldest first

Or GET /api/v1/admin/keeper/requests/{requestId}/events (ADMIN+). A request belonging to another workspace returns an empty list rather than a 404, so the endpoint cannot be used to probe which request ids exist.

Requests raised before the ledger migration have their current state backfilled (a PENDING at created_at plus the decision at decided_at) but no intermediate history — earlier rewrites, if any happened, are unrecoverable. That loss is exactly what the ledger stops from here on.

Audit Log Table Structure

CREATE TABLE audit_logs (
  id          TEXT PRIMARY KEY,
  workspace_id TEXT NOT NULL,
  user_id     TEXT,
  action      TEXT NOT NULL,
  entity_type TEXT NOT NULL,
  entity_id   TEXT,
  metadata    TEXT,          -- JSON object
  ip_address  TEXT,
  user_agent  TEXT,
  created_at  TEXT NOT NULL   -- ISO 8601
);

Column	Type	Description
`id`	`TEXT`	Random hex ID (32 chars)
`workspace_id`	`TEXT`	Workspace the action occurred in
`user_id`	`TEXT`	User who performed the action (null for system actions)
`action`	`TEXT`	Action type — CRUD verbs (`create`, `update`, `delete`) for agent mutations, `backup.*` for backup lifecycle events
`entity_type`	`TEXT`	Entity type (AGENT, CREW, MISSION, etc.)
`entity_id`	`TEXT`	ID of the affected entity
`metadata`	`TEXT`	JSON object with action-specific details
`ip_address`	`TEXT`	Client IP address
`user_agent`	`TEXT`	Client user agent string
`created_at`	`TEXT`	ISO 8601 timestamp

The audit log is append-only. There is no API or mechanism to delete or modify audit log entries. This ensures an immutable record for security review.

Querying the Audit Log

GET /api/v1/audit

Required role: OWNER or ADMIN (manage permission)

Query Parameters

Parameter	Type	Default	Description
`page`	`integer`	`1`	Page number (1-based)
`limit`	`integer`	`50`	Entries per page (1-100)
`action`	`string`		Filter by action (e.g., `create`, `delete`)
`entity_type`	`string`		Filter by entity type (e.g., `AGENT`, `CREW`)
`entity_id`	`string`		Filter by specific entity ID
`user_id`	`string`		Filter by user who performed the action
`date_from`	`string`		ISO 8601 start date
`date_to`	`string`		ISO 8601 end date
`search`	`string`		Free-text match against the acting user’s email / full name (joined from the `users` table)
`source`	`string`	`workspace`	Which trail to read: `workspace`, `crews`, `credentials`, `keeper`

Response

{
  "data": [
    {
      "id": "a1b2c3d4...",
      "workspace_id": "ws-uuid",
      "user_id": "user-uuid",
      "action": "create",
      "entity_type": "AGENT",
      "entity_id": "agent-uuid",
      "metadata": "{\"name\":\"Viktor\",\"role\":\"AGENT\"}",
      "ip_address": "192.168.1.100",
      "user_agent": "Mozilla/5.0...",
      "created_at": "2025-01-15T10:30:00Z",
      "user_email": "alice@example.com",
      "user_name": "Alice Chen"
    }
  ],
  "pagination": {
    "page": 1,
    "limit": 50,
    "total": 234,
    "total_pages": 5
  }
}

The response includes user_email and user_name joined from the users table, and entity_name — the display name of whatever the row touched, resolved per entity_type because entity_id is polymorphic. It is null when the target has no name to give (a backup path, a hard-deleted row), in which case the id is all there is. entity_name is deliberately not filtered on deleted_at: “who deleted Riley” is a question asked after Riley is gone, and that is the one place the log has to keep speaking.

Sources

A workspace keeps four audit trails, in four tables, on purpose — the keeper ledger is append-only and merging it into a general log would cost exactly the guarantee it exists for. Only the reading is unified: each source is projected onto the response shape above.

`source`	Table	Answers
`workspace` (default)	`audit_logs`	Settings, people, crews, credentials, agents, backups
`crews`	`crew_audit_log`	Cross-crew dispatch, messages, shared files
`credentials`	`credential_audit`	Which secret was used, revealed or rotated
`keeper`	`keeper_request_events`	Every gatekeeper decision, append-only

action, entity_type, entity_id, user_id and search apply to the workspace source only — the other trails index by their own shape, and a filter the server would silently ignore is worse than one that is not offered. page, limit and the date range mean the same thing everywhere. From the CLI:

crewship audit --source keeper --lines 100
crewship audit --source credentials --since 24h

Pagination

The audit log uses offset-based pagination:

Default limit: 50 entries per page
Maximum limit: 100 entries per page
Minimum limit: 1 entry per page
Pages are 1-based (page 1 is the first page)

Example Queries

All agent deletions
Actions by a specific user
Recent activity (date range)

GET /api/v1/audit?action=delete&entity_type=AGENT

GET /api/v1/audit?user_id=user-uuid

GET /api/v1/audit?date_from=2025-01-01T00:00:00Z&date_to=2025-01-31T23:59:59Z

How Audit Entries Are Written

Audit entries are written synchronously during mutation handlers using the WriteAuditLog function (defined at internal/api/internal_handler.go:62):

func WriteAuditLog(
    ctx context.Context,
    db *sql.DB,
    j journal.Emitter,
    action, entityType, entityID, userID, workspaceID string,
    metadata map[string]interface{},
)

Typical call site:

WriteAuditLog(r.Context(), h.db, h.journal, "create", "AGENT", agentID, userID, workspaceID, map[string]interface{}{
    "name": agent.Name,
    "role": agent.Role,
})

The function generates a random hex ID, writes the row in the same request context, and — when j is non-nil — dual-emits a typed audit.entity_* entry into the unified Crew Journal. If the write fails, a warning is logged but the original operation is not rolled back — the audit log is best-effort to avoid blocking user operations.

Tamper-Evidence: the Journal Hash-Chain

Crewship deliberately chooses accountability ex-post over real-time control, which makes the journal the load-bearing safety mechanism. To keep that record trustworthy, every journal_entries row is linked into a per-workspace hash-chain:

seq — a per-workspace monotonic sequence number (1-based). It gives a deterministic order independent of the random id primary key and the wall-clock ts (which can collide), and a deleted middle row shows up as a gap. A UNIQUE(workspace_id, seq) index enforces that no two rows share a slot.
prev_hash — the entry_hash of the immediately preceding entry in the same workspace. The first entry (genesis) uses the empty string.
entry_hash — a keyed HMAC-SHA256 over the entry’s committed content plus prev_hash.

The chain is keyed — that’s the point

The threat model is an attacker (or compromised operator) with write access to the database. A bare hash would give them no trouble: they could edit a row, recompute its entry_hash, and recompute prev_hash/entry_hash for every following row, producing a perfectly self-consistent forged chain. Crewship defeats this by keying the chain. entry_hash is HMAC-SHA256(chain_key, …), where chain_key is derived from the persisted ENCRYPTION_KEY (domain-separated, one-way) and never stored in the database:

chain_key = HMAC-SHA256(ENCRYPTION_KEY, "crewship journal chain v1\x00")

An attacker with DB write can still mutate rows, but cannot recompute a valid entry_hash without the key — so verification fails. The same key derivation is used by the write path, the verifier, and the upgrade migration’s backfill, so they never drift.

What the chain detects

Mutation — editing any committed field of a written row (the recomputed keyed hash no longer matches the stored one).
Reorder — swapping entries in place (the prev_hash linkage no longer matches).
Mid-chain deletion — removing a row leaves a seq gap with no signed checkpoint covering it.
Forged rewrite by a DB-write attacker — recomputing the hash columns without the key does not validate (this is the property a bare hash lacked).
Unrecorded priority change — the one committed field that is legitimately mutable is reconciled against its append-only change ledger, so flipping it directly in the DB is caught (see below).

Keeper decisions are inside this chain. /keeper/request and /keeper/execute both emit a keeper.decision entry, and keeper.decision is not in the compactor’s allowlist, so those entries are never rolled up and stay verifiable for their whole retention. The execute entry carries the command and its exit code, so it remains meaningful even if the keeper_requests row is later pruned.

Legitimate deletion vs. tampering: signed checkpoints

Some deletions are legitimate: the daily compaction worker rolls up high-volume low-signal entries and deletes the originals, and the pipeline-resurrect purge removes a defunct routine’s journal rows. These delete rows from the middle of the chain, which would otherwise read as tampering. To distinguish the two, every such delete writes a signed checkpoint into journal_chain_checkpoints in the same transaction as the delete. The checkpoint commits — under the same HMAC key — to the exact (seq, entry_hash) of every row it removed. Verification bridges a seq gap when a valid checkpoint covers it, and continues linking prev_hash pointers across the gap. An attacker cannot abuse this: forging a checkpoint requires the key, and a mid-chain delete with no checkpoint (or a checkpoint with a bad MAC) still fails verification.

Legitimate mutation vs. tampering: entry priority

One committed field is meant to change after write. An OWNER/ADMIN can mark an entry high, pin or permanent (POST /api/v1/journal/{id}/priority) — a load-bearing marker, since permanent entries are never compacted and pins land in the curated pins.md. Hashing that mutable column would make every authorised pin a permanent false “tampered” verdict, which is worse than no verification at all: it trains operators to ignore the result. So the chain commits to priority_at_emit, written once at emit and never updated, while the live priority column stays where every reader looks. The mutable column is not left unguarded. Each edit appends to journal_entry_priorities — also append-only by trigger — in the same transaction as the column update, and verification reconciles the two: the live value must be reachable from priority_at_emit by following the recorded chain of changes. That makes two distinct attacks visible:

a silent column flip (a raw DB write) leaves no ledger row at all; and
a fabricated ledger row whose previous_priority does not chain back to the emit-time value.

Why it matters that this is checked: downgrading a permanent entry to normal would let the next compaction pass remove it legitimately, complete with a valid signed checkpoint — laundering a deletion through a mechanism designed to authorise one.

This check bounds forgery rather than eliminating it. An attacker with DB write can append a fully self-consistent chain of fake edits. Three things stand in the way: the honest path is now verifiable (no false positives to hide behind); every real edit also emits a memory.priority_changed entry into the keyed chain, so a forged ledger with no corresponding chained entry is detectable by comparing the two; and — since #1572 — a forged un-pin no longer achieves anything, because compaction refuses to delete an entry that was permanent at emit regardless of what the ledger claims. The forgery can still mislead a reader of the priority column; it can no longer get the record destroyed.

Entries pinned before the upgrade keep working: priority_at_emit is seeded from their current priority and the edit ledger is deliberately left empty for them, so reconciliation holds with zero recorded changes. The pre-upgrade edit history is unrecoverable, and inventing a change nobody can be attributed to would be a fabricated audit record. The guarantee starts at the upgrade and runs forward.

Recovered is not resolved

An entry pinned before the upgrade is a partial exception to the note above. The backfill seeded priority_at_emit from the value at migration time, which for an already-edited row is not the value its stored hash was computed over — so the hash does not reproduce. Verification tries the four possible priorities against the stored HMAC and, when one reproduces it, reports the row as repairable: the content is proven authentic and the emit-time priority is recovered exactly. Producing content that hashes correctly under any candidate requires the key, so this removes a false positive without weakening the oracle. What it does not establish is where the live priority came from. A pre-v166 pin and an attacker’s write to both priority and priority_at_emit leave byte-identical rows. Recovery used to resolve that ambiguity silently in favour of “benign” and skip the live-priority check for such rows — which laundered a permanent → normal downgrade past the one check guarding compaction’s exemption (issue #1572). So a repairable row is now reported, not absolved:

it appears in repairable with the recovered emit-time value and as a priority break, so ok is false;
crewship journal verify prints it in its own block and exits non-zero;
compaction refuses to delete it (below).

The cost is that a genuine pre-upgrade pin reads red until an operator looks at it — a row which, before the chain moved off the mutable column, was a hard break anyway. Reporting a real ambiguity is the honest answer; resolving it in the attacker’s favour was not.

Compaction only destroys what it can prove

Detecting the downgrade is necessary but not sufficient: verification is read by an operator occasionally, while compaction runs on a timer and reads nothing. So the refusal also lives at the point of deletion. Before a compaction pass deletes anything it re-derives each candidate’s keyed hash, and it skips any row that:

does not reproduce its stored hash (tampered, or repairable);
is in the chain (seq > 0) but carries no stored hash;
has a live priority not reachable from the emit-time value through the ledger; or
was permanent at emit — whatever the live column says today.

The last rule is the one that survives future refactoring. permanent at emit is a keyed, unforgeable fact; the un-pin that moves the live column is a ledger row anyone with database write can append. An unauthenticated claim may move a label, but it does not authorise destroying the record the label protects. Un-pinning still works for curation — it just no longer makes the entry disposable by a timer. Rows that predate the hash-chain (seq = 0) make no tamper-evidence claim and are unaffected.

Residual gaps (no overclaiming). Two attacks are not fully covered: tail truncation — deleting the newest N entries leaves a shorter but internally-consistent chain (anchoring the tip with a signed high-water checkpoint is the tracked follow-up, issue #1369); and plaintext dev mode — when no ENCRYPTION_KEY is configured the chain key is derived from an empty seed and is therefore reproducible, so keying degrades to detecting only key-unaware edits. Production installs always have a persisted ENCRYPTION_KEY.

The audit log outlives what it refers to

A journal entry points at the crew, agent and mission it describes. Until schema v167 those pointers were ordinary foreign keys, and they gave the database itself the ability to edit and delete the audit log:

column	old constraint	what happened
`crew_id`	`REFERENCES crews(id) ON DELETE CASCADE`	deleting a crew deleted its audit rows
`agent_id`	`REFERENCES agents(id) ON DELETE CASCADE`	deleting an agent deleted its audit rows
`mission_id`	`REFERENCES missions(id) ON DELETE SET NULL`	deleting a mission rewrote `mission_id` to NULL on every row that referenced it

mission_id is one of the fields the chain hash commits to, so the last one silently broke verification on rows nobody had touched: the stored entry_hash had been computed over the real id. To the verifier, a foreign key doing its job and an attacker editing rows are the same event — it reported entry was modified after write. The other two were worse in kind, if quieter: they destroyed audit history outright, leaving a seq gap that reads as a malicious mid-chain deletion. All three are now plain TEXT columns. An entry keeps the ids it was written with even after the crew, agent or mission is gone — which is the point of an audit log. workspace_id deliberately keeps ON DELETE CASCADE: the chain is per-workspace, so a workspace’s journal going away with the workspace is coherent (and is what makes workspace teardown and backup --replace work).

Upgrading repairs the damage rather than papering over it. Values the old constraint nulled are recoverable from the entry’s refs JSON, which the constraint could not reach — but the migration does not trust that blindly. Each candidate is re-hashed and written back only when it reproduces the entry_hash already on disk, which (the hash being an HMAC under a key that is not in the database) proves it is the value the row was written with. Rows that cannot be proven are counted and logged, never rehashed: recomputing a hash over content nobody can vouch for would launder real tampering into a clean chain. Restoring a backup taken before the upgrade runs the same repair, so a restore cannot reintroduce the breakage.

Determinism

Hashing never depends on Go map iteration order. payload and refs are hashed as their stored JSON strings (encoding/json emits map keys sorted), and every field is length-framed (an 8-byte length prefix) before hashing, so no value can be confused with a delimiter or spill into its neighbour.

Verifying integrity

Run verification for the current workspace with the CLI (ADMIN or OWNER):

crewship journal verify

It walks the keyed chain and prints Journal chain OK — N entries verified against the keyed HMAC chain (plus a count of signed compaction checkpoints bridged, if any) or, on a break, the first bad seq/entry and the reason. The command exits non-zero when the chain is broken, so it can gate a cron job or CI check. The --format json variant emits the raw result for machine consumers. The same check is available over HTTP:

GET /api/v1/admin/journal/verify        # ADMIN or OWNER

{ "workspace_id": "ws-uuid", "ok": true, "count": 128, "checkpoints": 2 }

checkpoints is the number of valid signed compaction checkpoints applied while walking the chain. On a broken chain the response stays 200 with "ok": false plus broken_seq, broken_id, and a human-readable reason.

Upgrading an existing instance runs a migration that backfills every historical row into a valid chain (ordered by workspace_id, ts, id), so verification passes immediately after upgrade. On shared dev slots a nuke+reseed is still the cleanest path, since any out-of-band edits made before the upgrade are frozen into the chain as genuine.

Get Started

Guides

Security

Configuration

Manifest Reference

Audit Log

Audit Log

What Is Audited

Keeper Decision Auditing

The decision history is append-only

Audit Log Table Structure

Querying the Audit Log

GET /api/v1/audit

Query Parameters

Response

Sources

Example Queries

How Audit Entries Are Written

Tamper-Evidence: the Journal Hash-Chain

The chain is keyed — that’s the point

What the chain detects

Legitimate deletion vs. tampering: signed checkpoints

Legitimate mutation vs. tampering: entry priority

Recovered is not resolved

Compaction only destroys what it can prove

The audit log outlives what it refers to

Determinism

Verifying integrity

What’s Next

Admin API

RBAC

​Audit Log

​What Is Audited

​Keeper Decision Auditing

​The decision history is append-only

​Audit Log Table Structure

​Querying the Audit Log

​GET /api/v1/audit

​Query Parameters

​Response

​Sources

​Pagination

​Example Queries

​How Audit Entries Are Written

​Tamper-Evidence: the Journal Hash-Chain

​The chain is keyed — that’s the point

​What the chain detects

​Legitimate deletion vs. tampering: signed checkpoints

​Legitimate mutation vs. tampering: entry priority

​Recovered is not resolved

​Compaction only destroys what it can prove

​The audit log outlives what it refers to

​Determinism

​Verifying integrity

​What’s Next

Admin API

RBAC

Audit Log

What Is Audited

Keeper Decision Auditing

The decision history is append-only

Audit Log Table Structure

Querying the Audit Log

GET /api/v1/audit

Query Parameters

Response

Sources

Pagination

Example Queries

How Audit Entries Are Written

Tamper-Evidence: the Journal Hash-Chain

The chain is keyed — that’s the point

What the chain detects

Legitimate deletion vs. tampering: signed checkpoints

Legitimate mutation vs. tampering: entry priority

Recovered is not resolved

Compaction only destroys what it can prove

The audit log outlives what it refers to

Determinism

Verifying integrity

What’s Next