Internal IPC
Routes mounted under/api/v1/internal/* are not part of the public API. They are the surface used by the sidecar (running as UID 1002 inside every crew container) and a small set of orchestrator helpers. The sidecar holds the only internal-API credential inside a container — a workspace-bound token derived from CREWSHIP_INTERNAL_TOKEN (never the master itself); agents (UID 1001) cannot read it from disk or process memory.
This page is here so integrators building their own sidecar image, runtime adapter, or replay tool understand the contract. End users should never call these endpoints directly.
Every route below is wrapped in
internalAuth and authenticated with the X-Internal-Token header. Sidecars carry a workspace-bound token (derived per workspace from the master at sidecar start) that only authorizes their own workspace; the token lives only inside the sidecar (UID 1002); agents (UID 1001) cannot read it. These endpoints are not for external clients.Endpoints
One row per documented internal route, grouped by family. See Route catalog for the per-family detail and Authentication for the trust model.Authentication
Every internal route is wrapped ininternalAuth:
internal_token: in the config file or the CREWSHIP_INTERNAL_TOKEN environment variable wins; otherwise the master is derived deterministically from the persisted ENCRYPTION_KEY (HMAC-SHA256(ENCRYPTION_KEY, "crewship internal-token master derivation v1")). Because ENCRYPTION_KEY is stable across restarts (bootstrapped once to <dataDir>/secrets.env, or operator-supplied), the master is stable across restarts too — a crewshipd restart no longer invalidates the crew-bound tokens held by agent containers that outlived the process (#1385). Only when no encryption key is configured at all (plaintext dev mode) does the master fall back to a per-boot random.
For multi-host deployments, still set CREWSHIP_INTERNAL_TOKEN (or a shared ENCRYPTION_KEY) explicitly on every host, so a sidecar request that lands on a different host than the one that minted its token still validates.
Sidecars never receive the master token. At sidecar start, the orchestrator derives a workspace-bound token — wsv1.<workspace_id>.<hex(HMAC-SHA256(master, derivation_context || workspace_id))> — and injects it via the sidecar’s stdin IPCConfig. The derivation_context is a fixed, versioned domain-separation string ("crewship internal-token workspace binding v1\x00"); the MAC is computed over that context concatenated with the workspace ID, not over the workspace ID alone. Use internaltoken.DeriveWorkspaceToken (in internal/auth/internaltoken) as the canonical implementation — a custom sidecar that hashes only the workspace ID will mint tokens that fail validation. The internalAuth middleware validates the binding on every request: the MAC is re-derived from the embedded workspace_id and the in-memory master. The binding is enforced as a mandatory request scope:
- a
?workspace_idquery parameter that disagrees with the token’s bound workspace is rejected with403before the handler runs; and - when the caller omits
?workspace_id, the middleware injects the bound workspace into the query — so every handler that filters by?workspace_id(/credentials,/agents/{id}/resolve,/chats/{id}/resolve,/crews, …) is tenant-scoped automatically. There is no “legacy unscoped” path for a bound token.
404, never mutated. Handlers scoped by a workspace_id carried only in the JSON body (/cost/record, /journal/emit, /pipelines/save, and the issue / mission / assignment / query / escalation create handlers and the confidence report) enforce the binding in-handler via assertInternalTokenWorkspace (403 on a foreign tenant), since the auth middleware cannot inspect bodies.
Crew-bound tokens (#1159). When a run belongs to a crew, the orchestrator mints a crew-bound token instead — crwv1.<workspace_id>.<crew_id>.<hex(HMAC-SHA256(master, "crewship internal-token crew binding v1\x00" || workspace_id || \x00 || crew_id))>, via internaltoken.DeriveCrewToken. It carries the workspace binding above (same ?workspace_id inject / 403-on-mismatch rules) plus a crew binding: requireInternal rejects a ?crew_id that disagrees with the token’s crew (403) and exposes the bound crew to handlers via context (InternalTokenCrewFromContext). GET /api/v1/internal/credentials scopes its metadata listing to that cryptographic crew in preference to the ?crew_id query — so a compromised agent can neither omit crew_id to see the workspace-wide list nor forge a sibling crew’s id to enumerate its credential metadata (the #1031 / #1159 leak). A crew-less run (no crew_id) falls back to the workspace-bound wsv1 token, keeping the crew-less workspace-wide behaviour the in-process TokenSyncer relies on.
Like the body-workspace routes above, handlers that take crew_id in the JSON body (cost record, journal emit, keeper request/execute, pipeline save/exec, assignment run, escalation, query, MCP tool-call recording) plus POST /api/v1/internal/agents enforce the crew binding in-handler via assertBoundCrewWorkspaceDB (#1186): a crew-bound token naming a sibling crew in the body is rejected with 403, so it can no longer attribute cost rows or journal entries to — or hire agents into — another crew. Workspace-bound (wsv1) and master tokens stay workspace-wide by design.
Derivation is stateless, so a derived token is valid for as long as the master is — which, with a stable ENCRYPTION_KEY (or an explicit CREWSHIP_INTERNAL_TOKEN), now spans restarts rather than rolling on every boot (#1385). A token only stops validating when the master actually changes: the operator rotates CREWSHIP_INTERNAL_TOKEN, or the encryption key it is derived from changes. The unbound master token remains valid for host-side trusted callers (chat bridge, LLM proxy monitor) that never enter a container — but only from a loopback origin: a master token arriving from a Docker-bridge / LAN IP is refused with 403 (capping the blast radius of a master copy leaked into a container). Set CREWSHIP_INTERNAL_ALLOW_ANY=true to relax both the loopback pin and the network gate when a reverse proxy rewrites RemoteAddr.
Workspace / crew / agent / mission scope is not taken from request bodies in a way that can cross tenants. The IPC layer reads it from the sidecar’s IPCConfig (set by the orchestrator at exec time and pinned to the container) and projects it onto the request, and the token binding constrains every handler to its workspace. An agent that captured the sidecar’s token cannot forge cross-tenant attribution: the token only authorizes the workspace it is cryptographically bound to.
Network origin & reverse proxies
Before the token compare, a network-origin gate refuses/api/v1/internal/* from any peer that is not loopback or a private/RFC1918 range (Docker bridge, on-prem LAN), returning 404 so a public scanner can’t even confirm the route exists. The intended boundary: only same-host processes and the sidecar (over the Docker bridge) reach this surface.
The whole prefix answers one 404 (#1501). A request under /api/v1/internal/ that the router would not dispatch gets the same 404 {"error":"Not Found"} the origin gate returns — same body, same Content-Type, no Allow header — whether the path does not exist, or exists with a different method. GET /api/v1/internal/keeper/request (a POST-only route) is therefore 404, not 405, for every caller including an authenticated one: the method check happens in net/http’s router, before any middleware could tell the two apart, so the only way to stop 405 from mapping the surface is to answer it uniformly. Build integration URLs from this page’s method column rather than probing — a wrong method is indistinguishable from a wrong path by design. Path-cleaning redirects are likewise not served on this prefix: a non-canonical spelling that would clean onto an internal route — //api/v1/internal/credentials, /api/v1//internal/credentials, /api/v1/./internal/credentials — gets the same 404, not the 307 Temporary Redirect to the cleaned path that net/http would otherwise answer with. Send the canonical path; sidecars build theirs from constants.
If you must proxy the internal surface, restore a real origin boundary with the application-level backstop:
-
CREWSHIP_INTERNAL_TRUSTED_PROXIES— a comma-separated list of proxy CIDRs / IPs (e.g.127.0.0.1,10.8.0.0/24). When the direct peer is one of these, crewshipd resolves the true client fromX-Forwarded-For(the rightmost hop not in the trusted set — the leftmost entries are attacker-controlled), or fromX-Real-IPif noX-Forwarded-Foris present (nginx’s single-IP default), and applies the origin gate to that. A header that IS present but unusable (empty / all-trusted / a non-IPX-Real-IP) is refused (fail-closed). These headers are honored only for a trusted-proxy peer — a client connecting directly can’t spoof them. The list is explicit and never auto-populated with private ranges (auto-trusting10.0.0.0/8on a LAN would let any host spoof the header); an all-zeros CIDR (0.0.0.0/0,::/0) is rejected outright, and the configured ranges are logged at startup so a typo is visible. Unset (default) = today’s behaviour: gate on the direct peer only. -
CREWSHIP_INTERNAL_ALLOW_ANY=true— the blunt bypass. Disables the origin gate and the master-token loopback pin entirely, leavingX-Internal-Tokenas the sole guard. Use only when you fully trust the network path.
Resolved exception: the keeper/* family and tenant isolation (PR-F24 closed; PR-F25 pending)
Resolved exception: the keeper/* family and tenant isolation (PR-F24 closed; PR-F25 pending)
The
/api/v1/internal/keeper/* family (skill-review, behavior, memory-health, negative-learning) reads body.workspace_id AND ctx.workspace_id (the latter set by the internalWsCtx middleware from the ?workspace_id query parameter). Historically this family carried a documented cross-tenant gap. Both halves are now closed:- Asymmetric forgery (workspace A in query, workspace B in body) is rejected at the handler boundary by the
assertBodyWorkspaceMatchesCtxhelper ininternal/api/keeper_phase2.go. - Symmetric forgery (caller picks one foreign workspace consistently across query + body) is closed by
PR-F24: theX-Internal-Tokena sidecar holds is bound to its workspace, and bothinternalAuthandinternalWsCtxreject a?workspace_idthat disagrees with the binding.
PR-F25, drops body.workspace_id from these four handlers entirely and derives workspace from context only — an architectural simplification, no longer a security gap.Route catalog
Sourced from the per-domaininternal/api/router_*.go files (split by domain in May 2026 — see router_internal.go for the internal IPC surface). Methods and paths are exact.
Cost ledger
Journal
Credentials
The
POST create/rotate routes are registered only when the public CredentialHandler is wired (router_internal.go:99-102); test routers that omit it skip the mirror.
Chats
Agents and crews
The
POST .../agents/hire route is registered only when the public AgentHandler is wired (router_internal.go:75-78); the nil-safe adapter returns 500 otherwise.
Pipelines, routines, and skills
These are the trusted forward-targets for agent-authored pipelines and the sidecar’s slash-action routes. Each dispatches into the same public handler the dashboard / CLI uses, with workspace + role context injected at the internal boundary.routines/schedules and the two read routes are registered only when the PipelineHandler is wired; skills/generate only when the SkillGenerateHandler is wired. pipelines/test_run / save / run and the two drafts/* routes are always registered on the PipelineHandler.
Runs
Since PR #234, runs are reconstructed from journal entries; these endpoints exist for transitional callers and emit a
run.* journal entry as a side effect. New integrations should emit journal entries directly via /api/v1/internal/journal/emit instead.
Memory
The host side of the agent memory contract.X-Internal-Token authenticates the sidecar; X-Acting-Agent-Slug narrows it to one agent and is re-proved against the token’s workspace/crew binding, so the header can only narrow the token’s authority, never widen it.
See Agent memory for the profiles and conflict vocabulary.
Crew messaging and files
Assignments and queries
Missions and issues
Keeper
Presenting a human approval. A request or execute retry may carry the id of
an escalation a person resolved as ALLOW. The approval binds to the same
agent, credential and request type — and, for
/execute, the identical
command — is single-use (races resolve to exactly one winner; the retry’s
own row is never spendable, so approvals cannot chain), expires after
15 minutes, and is spent before execution. Only a row resolved by a
person is consumable; an adjudicator/model ALLOW mints nothing. Binding
failures (internal/api/keeper_approval.go):
Keeper Phase 2 (aux-LLM evaluators, PR-C / PRD §6 F4). All four are wrapped in
internalAuth and internalWsCtx (which reads ?workspace_id= into the request context — see the Authentication known-exception note). They are always registered, returning a deterministic 503 (“evaluator not configured”) until the evaluators are wired (router_internal.go:177-202).
MCP audit
Pages
The body of
PUT .../data is an envelope, not the payload: workspace_id, crew_id,
agent_id and author_run_id are injected by the caller (the dispatcher for
the page.write verb, the sidecar for an agent), panel names which panel of
the page to write, state carries the producer’s own verdict (ok or
failed, absent meaning ok), and the payload rides in data. author_run_id is optional — an agent in a container has no run, and
the provenance column is nullable for exactly that caller.
Producer authority is checked here, not upstream: the panel’s declared
producer: must name the acting routine or agent, or a human must have issued
a produce grant covering the panel. A refusal writes a journal entry and
notifies the page owner. See Pages.
Agent notifications
An agent posting a status update to a Slack channel a human explicitly wired
it to — distinct from the outbound routing the category preference
matrix governs
(
agents.message is deliberately excluded from that matrix; the pairing
itself is the authorization, see below). Implementation:
internal/api/internal_notifications.go (AgentNotifyHandler).
Four gates, checked in order, so a rejected call is told which one it failed:
- Identity — the agent exists in the claimed workspace, and (for a crew-bound token) belongs to that crew. An agent cannot send as another agent.
- Pairing — the channel must be paired to this agent. Default-deny; a
workspace admin grants it with
crewship notifychannel agents allow <channel-id> --agent <agent-id>. - Rate — a per-agent token bucket (burst 3, refill 1/60s per agent×channel), tighter than the human-facing preference matrix’s, because an agent in a loop can call a tool far faster than a human clicks.
- Scrub — the title and body are redacted by the same secret scrubber that guards memory and tool output before the message leaves the instance.
Agent notifications: send
body itself is capped at 8 KiB.
Response:
200 OK
notification.delivered on success,
notification.failed (with the dispatch error) otherwise — attributed to the
agent as actor, so “who sent this to Slack?” has an answer on the Activity
timeline.
Errors:
Port expose
User-facing port-expose lifecycle endpoints live on the public API — see Port Expose.
Endpoint deep-dives
The routes above are catalog entries; the sections below give the full request/response contract for the ones with non-trivial bodies, trust models, or error matrices.Cost record
paymaster.Record (which inserts the cost_ledger row and emits llm.call + optionally cost.incurred), then paymaster.EnforceQuota (which emits budget.warning / budget.exceeded based on the parsed rate-limit headers).
Request body: body cap 16 KiB.
Fields the sidecar has no authority over are derived server-side:
Response:
202 Accepted
paymaster.EnforceQuota (best-effort — its result does not affect the response code, only the journal). The response body is intentionally minimal; the journal entries (llm.call, cost.incurred, optionally budget.warning / budget.exceeded) are where the operator-visible artifacts land.
When had_status_429=true, the response is still 202 — the ledger row was written and budget.exceeded was emitted. The caller (sidecar) is responsible for propagating the upstream’s 429 back to the agent; this endpoint does not echo the 429 because it succeeded at recording the cost.
Errors:
Ephemeral hire
/spawn flow proxies here. The HireInternalAdapter reads workspace_id from the query string (the sidecar attaches it), injects a MANAGER role into the request context, then calls the public AgentHandler.Hire path unchanged — so the per-crew autonomy policy gate, the crews.max_ephemeral_agents quota, the audit log, and the inbox emission all run exactly as they do for a human hire. The adapter deliberately does not inject a user_id; the resulting audit row is attributed to actor.system. (internal/api/internal_hire.go:45)
Request body (same shape as the public POST /api/v1/agents/hire):
workspace_id is not read from the body — it comes from the ?workspace_id= query parameter the sidecar attaches.
Response: mirrors the public Hire handler — 201 Created (live ephemeral), 202 Accepted (waiting on inbox approval), with a body carrying id, slug, status, ephemeral, expires_at, decision, and optionally inbox_item_id.
Errors: 400 (missing workspace_id query param, or missing/invalid body fields), 403 (strict crew rejected the hire), 404 (crew or template not found), 429 (per-crew ephemeral quota reached), 500 (adapter not configured / DB error).
Pipeline save
X-Internal-Token runs upstream; the handler trusts the caller’s claim about author identity from the body. It parses + validates the DSL, runs cross-crew reference checks and cycle detection over the workspace’s saved pipelines, then persists. (internal/api/pipelines_crud.go:634)
Request body:
Response:
201 Created — the saved pipeline object (toPipelineResponse), plus a trigger block (kind, schedule_id, cron_expr, timezone, enabled, first_fire_at, approval_required, inbox_item_id) when the save requested one. A risky definition lands as proposed with an inbox review item, exactly as on the user path.
Errors: 400 (invalid JSON, or missing workspace_id / slug / definition), 409 — either slug already exists in workspace, or the schedule-preset gate: when the new definition renames or adds a required input that an active plan’s preset no longer satisfies, the answer is 409 {"error", "hint", "schedule_conflict"} where schedule_conflict is {schedule_id, name, reason} so a client can open the named plan and repair its preset, and hint spells out the way out (send a trigger block with the plan’s inputs in the same save, repair the plan first, or disable it, publish, then re-enable). 422 (DSL parse / validation / cycle-detection failure, an invalid save_token, the test-run gate not satisfied — save requires a fresh, passing test_run (within 5 minutes) — or a trigger / activation that fails validation, which also rolls back the routine), 500 (DB error).
Pages: agent creation
save_page MCP tool forwards to — the
route that lets an agent build a Page’s metadata + panels from scratch,
distinct from PUT /api/v1/internal/pages/{page}/data, which only pushes one
panel’s payload onto a page a human already created. Mirrors
pipelines/save’s trust model as closely as the two domains allow: identity
is proven against the token binding (never trusted from the body beyond
that), the request is gated on the authoring crew’s autonomy_level, and it
reuses the exact validation/insert machinery POST /api/v1/pages (the public
create route) uses. (internal/api/pages_internal_save.go:90)
Request body: capped at pages.MaxSpecBytes + 8 KiB envelope slack.
Response:
201 Created — the created page document (same shape as
GET /api/v1/pages/{slug}).
Autonomy gate. page_create is evaluated against the acting crew (the
authoring crew, before any target_crew_slug delegation) before validation
runs, so a held request never has to pass full spec validation to learn it
will not be created:
- Held (autonomy below the level
page_createrequires): unlikecrew_create/agent_create, a page has no “created but inert” staging state — no status column, no.proposeddirectory — so a held request creates nothing at all. The response is403with{"error", "reason", "crew_id", "autonomy_level", "policy_action", "pending_review": true, "requested_name", "requested_slug"}, and a blocking, ADMIN-addressed inbox item records what was attempted. - Allowed: the page is created; at
fullautonomy a non-blocking inbox notice is also left (“worth a glance” even though nothing needs approval).
Pages: project source
page_project MCP tool forwards to: an
agent edits the source project of a custom Page application and builds it,
without ever publishing. The route is draft-only by construction — its private
principal is minted only after the internal token, the acting agent, the exact
owner crew and the page_create autonomy gate all agree, and no header or body
field can produce a user identity or a publication. A person publishes the
reviewed build through the editor or crewship page project publish.
(internal/api/pages_project_internal.go)
Request body: capped at 1 MiB plus the identity envelope.
A field sent with an operation it does not apply to is
400 (“fields do not apply to this project operation”).
init, save and build are gated on the acting crew’s autonomy_level for page_create; a held request is 403 and changes no draft and starts no build. read, status and check need the owning crew and a current agent in it. Every operation takes the Page’s lease, so an agent and the editor cannot write the same draft concurrently. The operator-side prerequisites (page_projects_path, Git, the pinned page_build_image) are in the YAML configuration and Pages apps operations.
Slash-action mirrors
Four internal routes mirror public handlers so the sidecar’s slash-action surface (and, where noted, autonomous agent tool calls) have a trusted backend to proxy into. Each adapter readsworkspace_id from the ?workspace_id= query parameter and injects a role into the request context before dispatching to the shared public handler. They follow a dual-path model (PRD-SLASH-CAPABILITIES-2026 §6.5):
- User-initiated (
X-Caller-User-Idheader present): the adapter gates on the caller’s per-action capability (routine.create,skill.create,credential.create,credential.rotate) and stamps the real user id for audit attribution. - Autonomous-agent (
X-Caller-User-Idabsent): skips the capability gate — a capability belongs to a user and this path has none — and is instead gated on the calling crew’sautonomy_level. See Autonomy gate on agent-driven creation.
Credentials are the exception: both credential mirrors reject with
401 when X-Caller-User-Id is absent — autonomous-agent credential mutation is intentionally not supported, because the public Create/Rotate handlers write a human user id into the audit / rotation-initiator columns and rotation has a workspace-wide blast radius. (internal/api/internal_credentials_mutate.go:99-113)
Request / response shapes match the underlying public endpoints:
routines/schedules→ body is the schedule shape (name,target_pipeline_slugortarget_pipeline_id,cron_expr,timezone,inputs,enabled);cron_expris required. Returns201 Createdwith the schedule object; belowfullthe schedule is created enabled and leaves a non-blocking inbox notice. Returns403atstrict. (A202 Acceptedwithenabled: falseis still possible when the policy resolver is unwired — the gate fails closed, and a suppliedenabled: trueis overridden on that path.) See Autonomy gate on agent-driven creation. (internal/api/internal_routines.go:76,internal/api/pipeline_schedules.go:82)skills/generate→ body{ "slug", "prompt", "model"? }(slug+promptrequired). The adapter stampsworkspace_idas the{workspaceId}path value the public handler expects. Returns200 OKwith{ "skill_id", "slug", "content", "scan_status", ... }atfullautonomy; below that the generated document is staged for review and the response is202 Acceptedwith{ "file_name", "slug", "scan_status", "pending_review": true }. Needs an AnthropicAPI_KEYcredential in the workspace (412otherwise). (internal/api/internal_skills.go:49,internal/api/skills_generate.go:94)credentials(create) → same body as Create Credential. Returns201 Created. (internal/api/internal_credentials_mutate.go:49)credentials/{credentialId}/rotate→ same body as Rotate Credential (valuerequired,grace_secondsoptional).credentialIdis read from the path. Returns200 OKwith the rotation object. (internal/api/internal_credentials_mutate.go:76)
400 (missing workspace_id query param or body validation), 401 (credentials only: missing X-Caller-User-Id), 403 (capability denied), 500 (adapter not configured).
Autonomy gate on agent-driven creation
Six internal routes let an agent create something that keeps acting after the request ends. Each one now consults the calling crew’sautonomy_level (see Policies) before writing, using the same decision matrix POST /api/v1/agents/hire already follows.
Two rows block below full and two do not, and the split is deliberate. crew_create and agent_create bring a durable principal into existence — something that will act again tomorrow, and in the agent’s case with a system_prompt the agent itself wrote. mission_create and routine_schedule_create create no principal, so below strict they proceed and leave a notice instead. Note this is oversight, not the security boundary: what stops an agent escaping a restrictive policy is that strict refuses outright and that a created crew inherits its creator’s autonomy_level rather than the schema default.
Refused (
403) returns a structured body — error, reason, crew_id, autonomy_level, policy_action — so the CLI can name the crewship policy set that would unblock it.
Held (202 Accepted) means the row is written but is inert until an operator approves it. The response carries decision, autonomy_level, pending_review: true and approval_id. What “inert” means per route:
The schedule row is no longer reached by any autonomy level —
strict refuses and everything below it proceeds. It is documented because the machinery is still live in two cases: a pending row written by an instance that ran an earlier build, and the fail-closed fallback when the policy resolver is unwired (which holds even the actions guided now allows). The mission row applies at strict only.kind=autonomy_gate, decidable with crewship approvals approve|deny <id> or POST /api/v1/approvals/{id}/decide, and are mirrored as a blocking, ADMIN-addressed inbox waitpoint. Denying leaves the artefact inert rather than deleting it, and so does letting the approval time out — the gate fails closed.
Staged is the skill variant: the SKILL.md is written to the crew’s .proposed directory behind a blocking review item and never reaches the live skills registry until crewship skill proposed approve promotes it. A crew-less (workspace-bound token) caller has no .proposed directory to stage into, so skills/generate refuses with 403 for that caller instead.
At full autonomy the creation proceeds immediately and leaves a non-blocking inbox notice — full autonomy is still autonomous, but a new crew, agent or cron entry is exactly the change an operator wants to see having happened.
Every arm — refused, held, or allowed — writes an audit entry carrying decision, autonomy_level and policy_action.
Related
- Paymaster guide — billing modes, quota enforcement.
- Crew Journal —
llm.call,budget.*entry types. - Architecture — IPC — Unix socket mechanics.