Skip to main content

​
Internal IPC

Routes mounted under /api/v1/internal/* are not part of the public API. They are the surface used by the sidecar (running as UID 1002 inside every crew container) and a small set of orchestrator helpers. The sidecar holds the only internal-API credential inside a container — a workspace-bound token derived from CREWSHIP_INTERNAL_TOKEN (never the master itself); agents (UID 1001) cannot read it from disk or process memory. This page is here so integrators building their own sidecar image, runtime adapter, or replay tool understand the contract. End users should never call these endpoints directly.
Every route below is wrapped in internalAuth and authenticated with the X-Internal-Token header. Sidecars carry a workspace-bound token (derived per workspace from the master at sidecar start) that only authorizes their own workspace; the token lives only inside the sidecar (UID 1002); agents (UID 1001) cannot read it. These endpoints are not for external clients.

​
Endpoints

One row per documented internal route, grouped by family. See Route catalog for the per-family detail and Authentication for the trust model.

​
Authentication

Every internal route is wrapped in internalAuth:
The master token resolves at server startup in this order: an explicit internal_token: in the config file or the CREWSHIP_INTERNAL_TOKEN environment variable wins; otherwise the master is derived deterministically from the persisted ENCRYPTION_KEY (HMAC-SHA256(ENCRYPTION_KEY, "crewship internal-token master derivation v1")). Because ENCRYPTION_KEY is stable across restarts (bootstrapped once to <dataDir>/secrets.env, or operator-supplied), the master is stable across restarts too — a crewshipd restart no longer invalidates the crew-bound tokens held by agent containers that outlived the process (#1385). Only when no encryption key is configured at all (plaintext dev mode) does the master fall back to a per-boot random. For multi-host deployments, still set CREWSHIP_INTERNAL_TOKEN (or a shared ENCRYPTION_KEY) explicitly on every host, so a sidecar request that lands on a different host than the one that minted its token still validates.
One-time upgrade step. The deploy that first introduces the stable master rotates it once — from the old per-boot random to the ENCRYPTION_KEY-derived value. A per-crew container that outlived that deploy still holds a token minted under the old master and is orphaned exactly once (it does not self-heal). Recreate the surviving containers (a normal redeploy / re-dispatch re-mints), or run crewship admin reap-orphan-containers --apply to clear just the orphaned ones. Every restart after this is a no-op for live containers. A running sidecar advertises a token_fp fingerprint of the crew-bound token it holds on its /health endpoint so the server can tell an orphaned container from a healthy one without ever seeing the token itself.
Sidecars never receive the master token. At sidecar start, the orchestrator derives a workspace-bound token — wsv1.<workspace_id>.<hex(HMAC-SHA256(master, derivation_context || workspace_id))> — and injects it via the sidecar’s stdin IPCConfig. The derivation_context is a fixed, versioned domain-separation string ("crewship internal-token workspace binding v1\x00"); the MAC is computed over that context concatenated with the workspace ID, not over the workspace ID alone. Use internaltoken.DeriveWorkspaceToken (in internal/auth/internaltoken) as the canonical implementation — a custom sidecar that hashes only the workspace ID will mint tokens that fail validation. The internalAuth middleware validates the binding on every request: the MAC is re-derived from the embedded workspace_id and the in-memory master. The binding is enforced as a mandatory request scope:
  • a ?workspace_id query parameter that disagrees with the token’s bound workspace is rejected with 403 before the handler runs; and
  • when the caller omits ?workspace_id, the middleware injects the bound workspace into the query — so every handler that filters by ?workspace_id (/credentials, /agents/{id}/resolve, /chats/{id}/resolve, /crews, …) is tenant-scoped automatically. There is no “legacy unscoped” path for a bound token.
Path-param mutations that don’t read the query (chat message-count / title, run finalize, credential status) additionally constrain their lookup by the bound workspace; a foreign-tenant row returns 404, never mutated. Handlers scoped by a workspace_id carried only in the JSON body (/cost/record, /journal/emit, /pipelines/save, and the issue / mission / assignment / query / escalation create handlers and the confidence report) enforce the binding in-handler via assertInternalTokenWorkspace (403 on a foreign tenant), since the auth middleware cannot inspect bodies. Crew-bound tokens (#1159). When a run belongs to a crew, the orchestrator mints a crew-bound token instead — crwv1.<workspace_id>.<crew_id>.<hex(HMAC-SHA256(master, "crewship internal-token crew binding v1\x00" || workspace_id || \x00 || crew_id))>, via internaltoken.DeriveCrewToken. It carries the workspace binding above (same ?workspace_id inject / 403-on-mismatch rules) plus a crew binding: requireInternal rejects a ?crew_id that disagrees with the token’s crew (403) and exposes the bound crew to handlers via context (InternalTokenCrewFromContext). GET /api/v1/internal/credentials scopes its metadata listing to that cryptographic crew in preference to the ?crew_id query — so a compromised agent can neither omit crew_id to see the workspace-wide list nor forge a sibling crew’s id to enumerate its credential metadata (the #1031 / #1159 leak). A crew-less run (no crew_id) falls back to the workspace-bound wsv1 token, keeping the crew-less workspace-wide behaviour the in-process TokenSyncer relies on. Like the body-workspace routes above, handlers that take crew_id in the JSON body (cost record, journal emit, keeper request/execute, pipeline save/exec, assignment run, escalation, query, MCP tool-call recording) plus POST /api/v1/internal/agents enforce the crew binding in-handler via assertBoundCrewWorkspaceDB (#1186): a crew-bound token naming a sibling crew in the body is rejected with 403, so it can no longer attribute cost rows or journal entries to — or hire agents into — another crew. Workspace-bound (wsv1) and master tokens stay workspace-wide by design. Derivation is stateless, so a derived token is valid for as long as the master is — which, with a stable ENCRYPTION_KEY (or an explicit CREWSHIP_INTERNAL_TOKEN), now spans restarts rather than rolling on every boot (#1385). A token only stops validating when the master actually changes: the operator rotates CREWSHIP_INTERNAL_TOKEN, or the encryption key it is derived from changes. The unbound master token remains valid for host-side trusted callers (chat bridge, LLM proxy monitor) that never enter a container — but only from a loopback origin: a master token arriving from a Docker-bridge / LAN IP is refused with 403 (capping the blast radius of a master copy leaked into a container). Set CREWSHIP_INTERNAL_ALLOW_ANY=true to relax both the loopback pin and the network gate when a reverse proxy rewrites RemoteAddr. Workspace / crew / agent / mission scope is not taken from request bodies in a way that can cross tenants. The IPC layer reads it from the sidecar’s IPCConfig (set by the orchestrator at exec time and pinned to the container) and projects it onto the request, and the token binding constrains every handler to its workspace. An agent that captured the sidecar’s token cannot forge cross-tenant attribution: the token only authorizes the workspace it is cryptographically bound to.

​
Network origin & reverse proxies

Before the token compare, a network-origin gate refuses /api/v1/internal/* from any peer that is not loopback or a private/RFC1918 range (Docker bridge, on-prem LAN), returning 404 so a public scanner can’t even confirm the route exists. The intended boundary: only same-host processes and the sidecar (over the Docker bridge) reach this surface. The whole prefix answers one 404 (#1501). A request under /api/v1/internal/ that the router would not dispatch gets the same 404 {"error":"Not Found"} the origin gate returns — same body, same Content-Type, no Allow header — whether the path does not exist, or exists with a different method. GET /api/v1/internal/keeper/request (a POST-only route) is therefore 404, not 405, for every caller including an authenticated one: the method check happens in net/http’s router, before any middleware could tell the two apart, so the only way to stop 405 from mapping the surface is to answer it uniformly. Build integration URLs from this page’s method column rather than probing — a wrong method is indistinguishable from a wrong path by design. Path-cleaning redirects are likewise not served on this prefix: a non-canonical spelling that would clean onto an internal route — //api/v1/internal/credentials, /api/v1//internal/credentials, /api/v1/./internal/credentials — gets the same 404, not the 307 Temporary Redirect to the cleaned path that net/http would otherwise answer with. Send the canonical path; sidecars build theirs from constants.
A public reverse proxy erases the origin gate. If Caddy/nginx terminates public traffic and proxies /api/v1/internal/* to crewshipd, the direct peer crewshipd sees is the proxy (typically 127.0.0.1), so every off-host request sails through the network gate and the shared X-Internal-Token becomes the only guard (#1020). The real fix is to not expose the internal surface at all — do one of:
  • Deny it at the proxy (recommended). Caddy:
    nginx:
  • Bind crewshipd’s internal listener to a private interface / unix socket and point only the sidecar at it, keeping the public vhost on a separate address.
If you must proxy the internal surface, restore a real origin boundary with the application-level backstop:
  • CREWSHIP_INTERNAL_TRUSTED_PROXIES — a comma-separated list of proxy CIDRs / IPs (e.g. 127.0.0.1,10.8.0.0/24). When the direct peer is one of these, crewshipd resolves the true client from X-Forwarded-For (the rightmost hop not in the trusted set — the leftmost entries are attacker-controlled), or from X-Real-IP if no X-Forwarded-For is present (nginx’s single-IP default), and applies the origin gate to that. A header that IS present but unusable (empty / all-trusted / a non-IP X-Real-IP) is refused (fail-closed). These headers are honored only for a trusted-proxy peer — a client connecting directly can’t spoof them. The list is explicit and never auto-populated with private ranges (auto-trusting 10.0.0.0/8 on a LAN would let any host spoof the header); an all-zeros CIDR (0.0.0.0/0, ::/0) is rejected outright, and the configured ranges are logged at startup so a typo is visible. Unset (default) = today’s behaviour: gate on the direct peer only.
    Your proxy must forward the client — set X-Forwarded-For (proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;) or X-Real-IP. If it forwards neither, crewshipd treats the request as a same-host self-call and gates on the proxy’s own (loopback/private) IP — i.e. it fails OPEN, exactly the #1020 hole, so every off-host request would pass the origin gate. The self-call fallback exists so crewshipd’s in-process loopback calls keep working; it is not a safety net for a misconfigured proxy.
  • CREWSHIP_INTERNAL_ALLOW_ANY=true — the blunt bypass. Disables the origin gate and the master-token loopback pin entirely, leaving X-Internal-Token as the sole guard. Use only when you fully trust the network path.
The /api/v1/internal/keeper/* family (skill-review, behavior, memory-health, negative-learning) reads body.workspace_id AND ctx.workspace_id (the latter set by the internalWsCtx middleware from the ?workspace_id query parameter). Historically this family carried a documented cross-tenant gap. Both halves are now closed:
  • Asymmetric forgery (workspace A in query, workspace B in body) is rejected at the handler boundary by the assertBodyWorkspaceMatchesCtx helper in internal/api/keeper_phase2.go.
  • Symmetric forgery (caller picks one foreign workspace consistently across query + body) is closed by PR-F24: the X-Internal-Token a sidecar holds is bound to its workspace, and both internalAuth and internalWsCtx reject a ?workspace_id that disagrees with the binding.
The remaining cleanup, PR-F25, drops body.workspace_id from these four handlers entirely and derives workspace from context only — an architectural simplification, no longer a security gap.

​
Route catalog

Sourced from the per-domain internal/api/router_*.go files (split by domain in May 2026 — see router_internal.go for the internal IPC surface). Methods and paths are exact.

​
Cost ledger

​
Journal

​
Credentials

The POST create/rotate routes are registered only when the public CredentialHandler is wired (router_internal.go:99-102); test routers that omit it skip the mirror.

​
Chats

​
Agents and crews

The POST .../agents/hire route is registered only when the public AgentHandler is wired (router_internal.go:75-78); the nil-safe adapter returns 500 otherwise.

​
Pipelines, routines, and skills

These are the trusted forward-targets for agent-authored pipelines and the sidecar’s slash-action routes. Each dispatches into the same public handler the dashboard / CLI uses, with workspace + role context injected at the internal boundary. routines/schedules and the two read routes are registered only when the PipelineHandler is wired; skills/generate only when the SkillGenerateHandler is wired. pipelines/test_run / save / run are always registered on the PipelineHandler.
Why the read tools need an internal route at all (#1763). The sidecar authenticates with X-Internal-Token, and extractToken reads only Authorization: Bearer and the session cookies. list_routines and discover_capabilities forwarded to the public, JWT-authed routes, so both answered 401 — every in-container agent asked to author a routine could see neither what its crew can reach nor what routines already existed, and wrote from memory instead. save_routine and run_routine were never affected because they already targeted this surface.The fix is a second door onto the same handlers, not teaching extractToken about the internal header: that would hand every holder of the shared internal secret a user-equivalent session across the whole public API — a far larger decision than two tools being unreachable. TestPublicPipelineRoutes_RejectTheInternalToken pins the public routes’ refusal so a future change cannot quietly widen it.

​
Runs

Since PR #234, runs are reconstructed from journal entries; these endpoints exist for transitional callers and emit a run.* journal entry as a side effect. New integrations should emit journal entries directly via /api/v1/internal/journal/emit instead.

​
Crew messaging and files

​
Assignments and queries

​
Missions and issues

​
Keeper

Keeper Phase 2 (aux-LLM evaluators, PR-C / PRD §6 F4). All four are wrapped in internalAuth and internalWsCtx (which reads ?workspace_id= into the request context — see the Authentication known-exception note). They are always registered, returning a deterministic 503 (“evaluator not configured”) until the evaluators are wired (router_internal.go:177-202).

​
MCP audit

​
Pages

The body is an envelope, not the payload: workspace_id, crew_id, agent_id and author_run_id are injected by the caller (the dispatcher for the page.write verb, the sidecar for an agent), panel names which panel of the page to write, state carries the producer’s own verdict (ok or failed, absent meaning ok), and the payload rides in data. author_run_id is optional — an agent in a container has no run, and the provenance column is nullable for exactly that caller. Producer authority is checked here, not upstream: the panel’s declared producer: must name the acting routine or agent, or a human must have issued a produce grant covering the panel. A refusal writes a journal entry and notifies the page owner. See Pages.

​
Port expose

User-facing port-expose lifecycle endpoints live on the public API — see Port Expose.

​
Endpoint deep-dives

The routes above are catalog entries; the sections below give the full request/response contract for the ones with non-trivial bodies, trust models, or error matrices.

​
Cost record

The sidecar’s write target after parsing an LLM response. Validates the request, calls paymaster.Record (which inserts the cost_ledger row and emits llm.call + optionally cost.incurred), then paymaster.EnforceQuota (which emits budget.warning / budget.exceeded based on the parsed rate-limit headers). Request body: body cap 16 KiB.
Fields the sidecar has no authority over are derived server-side: Response: 202 Accepted
The handler writes the ledger row synchronously, then runs paymaster.EnforceQuota (best-effort — its result does not affect the response code, only the journal). The response body is intentionally minimal; the journal entries (llm.call, cost.incurred, optionally budget.warning / budget.exceeded) are where the operator-visible artifacts land. When had_status_429=true, the response is still 202 — the ledger row was written and budget.exceeded was emitted. The caller (sidecar) is responsible for propagating the upstream’s 429 back to the agent; this endpoint does not echo the 429 because it succeeded at recording the cost. Errors:

​
Ephemeral hire

The internal entry for a LEAD agent spawning a short-lived “contractor” agent. The sidecar’s /spawn flow proxies here. The HireInternalAdapter reads workspace_id from the query string (the sidecar attaches it), injects a MANAGER role into the request context, then calls the public AgentHandler.Hire path unchanged — so the per-crew autonomy policy gate, the crews.max_ephemeral_agents quota, the audit log, and the inbox emission all run exactly as they do for a human hire. The adapter deliberately does not inject a user_id; the resulting audit row is attributed to actor.system. (internal/api/internal_hire.go:45) Request body (same shape as the public POST /api/v1/agents/hire): workspace_id is not read from the body — it comes from the ?workspace_id= query parameter the sidecar attaches. Response: mirrors the public Hire handler — 201 Created (live ephemeral), 202 Accepted (waiting on inbox approval), with a body carrying id, slug, status, ephemeral, expires_at, decision, and optionally inbox_item_id. Errors: 400 (missing workspace_id query param, or missing/invalid body fields), 403 (strict crew rejected the hire), 404 (crew or template not found), 429 (per-crew ephemeral quota reached), 500 (adapter not configured / DB error).

​
Pipeline save

The trusted endpoint the sidecar forwards to when an agent emits a new pipeline definition. X-Internal-Token runs upstream; the handler trusts the caller’s claim about author identity from the body. It parses + validates the DSL, runs cross-crew reference checks and cycle detection over the workspace’s saved pipelines, then persists. (internal/api/pipelines_crud.go:634) Request body: Response: 201 Created — the saved pipeline object (toPipelineResponse). Errors: 400 (invalid JSON, or missing workspace_id / slug / definition), 409 (slug already exists in the workspace), 422 (DSL parse / validation / cycle-detection failure, or the test-run gate was not satisfied), 500 (DB error).

​
Slash-action mirrors

Four internal routes mirror public handlers so the sidecar’s slash-action surface (and, where noted, autonomous agent tool calls) have a trusted backend to proxy into. Each adapter reads workspace_id from the ?workspace_id= query parameter and injects a role into the request context before dispatching to the shared public handler. They follow a dual-path model (PRD-SLASH-CAPABILITIES-2026 §6.5):
  • User-initiated (X-Caller-User-Id header present): the adapter gates on the caller’s per-action capability (routine.create, skill.create, credential.create, credential.rotate) and stamps the real user id for audit attribution.
  • Autonomous-agent (X-Caller-User-Id absent): skips the capability gate — a capability belongs to a user and this path has none — and is instead gated on the calling crew’s autonomy_level. See Autonomy gate on agent-driven creation.
Credentials are the exception: both credential mirrors reject with 401 when X-Caller-User-Id is absent — autonomous-agent credential mutation is intentionally not supported, because the public Create/Rotate handlers write a human user id into the audit / rotation-initiator columns and rotation has a workspace-wide blast radius. (internal/api/internal_credentials_mutate.go:99-113) Request / response shapes match the underlying public endpoints:
  • routines/schedules → body is the schedule shape (name, target_pipeline_slug or target_pipeline_id, cron_expr, timezone, inputs, enabled); cron_expr is required. Returns 201 Created with the schedule object; below full the schedule is created enabled and leaves a non-blocking inbox notice. Returns 403 at strict. (A 202 Accepted with enabled: false is still possible when the policy resolver is unwired — the gate fails closed, and a supplied enabled: true is overridden on that path.) See Autonomy gate on agent-driven creation. (internal/api/internal_routines.go:76, internal/api/pipeline_schedules.go:82)
  • skills/generate → body { "slug", "prompt", "model"? } (slug + prompt required). The adapter stamps workspace_id as the {workspaceId} path value the public handler expects. Returns 200 OK with { "skill_id", "slug", "content", "scan_status", ... } at full autonomy; below that the generated document is staged for review and the response is 202 Accepted with { "file_name", "slug", "scan_status", "pending_review": true }. Needs an Anthropic API_KEY credential in the workspace (412 otherwise). (internal/api/internal_skills.go:49, internal/api/skills_generate.go:94)
  • credentials (create) → same body as Create Credential. Returns 201 Created. (internal/api/internal_credentials_mutate.go:49)
  • credentials/{credentialId}/rotate → same body as Rotate Credential (value required, grace_seconds optional). credentialId is read from the path. Returns 200 OK with the rotation object. (internal/api/internal_credentials_mutate.go:76)
Common errors across all four: 400 (missing workspace_id query param or body validation), 401 (credentials only: missing X-Caller-User-Id), 403 (capability denied), 500 (adapter not configured).

​
Autonomy gate on agent-driven creation

Six internal routes let an agent create something that keeps acting after the request ends. Each one now consults the calling crew’s autonomy_level (see Policies) before writing, using the same decision matrix POST /api/v1/agents/hire already follows. Two rows block below full and two do not, and the split is deliberate. crew_create and agent_create bring a durable principal into existence — something that will act again tomorrow, and in the agent’s case with a system_prompt the agent itself wrote. mission_create and routine_schedule_create create no principal, so below strict they proceed and leave a notice instead. Note this is oversight, not the security boundary: what stops an agent escaping a restrictive policy is that strict refuses outright and that a created crew inherits its creator’s autonomy_level rather than the schema default. Refused (403) returns a structured body — error, reason, crew_id, autonomy_level, policy_action — so the CLI can name the crewship policy set that would unblock it. Held (202 Accepted) means the row is written but is inert until an operator approves it. The response carries decision, autonomy_level, pending_review: true and approval_id. What “inert” means per route:
“Inert” is true for the doors an agent can reach by being named, not for every door in the product. Three surfaces still resolve an agent and run it without reading agents.status: /query (peer questions), the webhook trigger, and a routine’s agent step. All three predate the autonomy gate and none of them can be reached by mentioning an agent or assigning it work, which is why they are listed rather than claimed closed. The authoritative list lives beside the code, in internal/api/internal_status.go.POST /api/v1/agents/{agentId}/approve-hire is not a release path for a gate-held agent and answers 409. It decides ephemeral hire rows only. It was briefly widened to decide autonomy-gate rows too, which let a MANAGER release a hold the gate addresses to OWNER/ADMIN and let a terminal (denied or timed-out) row be released anyway; narrowing it back is what closed that. The one release surface is POST /api/v1/approvals/{id}/decide — crewship approvals approve <id>.
The schedule row is no longer reached by any autonomy level — strict refuses and everything below it proceeds. It is documented because the machinery is still live in two cases: a pending row written by an instance that ran an earlier build, and the fail-closed fallback when the policy resolver is unwired (which holds even the actions guided now allows). The mission row applies at strict only.
Holds land on the standard approvals queue as kind=autonomy_gate, decidable with crewship approvals approve|deny <id> or POST /api/v1/approvals/{id}/decide, and are mirrored as a blocking, ADMIN-addressed inbox waitpoint. Denying leaves the artefact inert rather than deleting it, and so does letting the approval time out — the gate fails closed. Staged is the skill variant: the SKILL.md is written to the crew’s .proposed directory behind a blocking review item and never reaches the live skills registry until crewship skill proposed approve promotes it. A crew-less (workspace-bound token) caller has no .proposed directory to stage into, so skills/generate refuses with 403 for that caller instead. At full autonomy the creation proceeds immediately and leaves a non-blocking inbox notice — full autonomy is still autonomous, but a new crew, agent or cron entry is exactly the change an operator wants to see having happened. Every arm — refused, held, or allowed — writes an audit entry carrying decision, autonomy_level and policy_action.