Skip to main content

Orchestration

Crewship’s orchestration system manages multi-agent missions through the MissionEngine (internal/orchestrator/mission.go). It handles task scheduling, dependency resolution, failure recovery, and cross-crew coordination.
“Orchestration” here means the engine subsystem in internal/orchestrator/, not a navigable page. After the Plan/Run/Build/System IA refactor, the user-facing surfaces are split: Routines for reusable recipes, Issues for the work-item tracker, Inbox for your actionable feed, and Activity for the live trace canvas. The legacy /orchestration route now soft-redirects to /activity.

Mission Lifecycle

A mission progresses through these states:
  1. PLANNING — Mission created, tasks defined (or waiting for Lead to plan)
  2. IN_PROGRESS — Tasks being scheduled and executed
  3. REVIEW — All tasks finished (none failed). The mission enters review before final completion, allowing humans to inspect results
  4. COMPLETED — Mission accepted after review
  5. FAILED — A task failed and could not recover, or deadlock/timeout detected
  6. CANCELLED — Manually stopped by user or system
The REVIEW state is inserted between IN_PROGRESS and COMPLETED. When all tasks reach a terminal state (COMPLETED, FAILED, or SKIPPED) and none have failed, the mission transitions to REVIEW rather than directly to COMPLETED. If any task failed, the mission transitions to FAILED instead.

The Mission Engine

The MissionEngine is the central orchestrator. Its behaviour is governed by a handful of fixed parameters:

Mission Loop

The runMissionLoop function runs as a goroutine for each active mission. Every 3 seconds it:

Restart Durability

Mission loops are in-memory goroutines, but missions survive server restarts. At boot — right after orphaned-run recovery — the server scans the database for missions still in IN_PROGRESS and re-attaches an orchestration loop to each one. Tasks the previous process never dispatched are picked up on the next tick, and BLOCKED tasks whose dependencies already completed are self-healed back to PENDING. No operator action is needed after a restart or crash; the 2-hour mission timeout restarts from the moment of re-attach.

Scaling / Concurrency

Separate from per-mission task scheduling above, the orchestrator bounds how many agent-run exec fan-outs (RunAgent) may be in flight across the whole server at once, via a semaphore (runSem in internal/orchestrator/orchestrator.go). Each run fans out roughly a dozen container execs — sidecar start, mkdir/manifest/credential setup, and the long-lived agent CLI exec — all against a single Docker daemon, so unbounded concurrency here is the fastest path to daemon saturation or host OOM. Precedence when both are set: CREWSHIP_MAX_CONCURRENT_RUNS (env) wins over orchestrator.max_concurrent_runs (config file), which wins over the built-in default of 8.
Tuning guidance:
  • Raising the cap increases parallelism, but the practical ceiling is whichever comes first: host RAM (each run’s container + agent CLI + MCP processes) or your LLM provider’s own rate limits.
  • Idle agents and provisioned-but-inactive crews cost nothing against this cap — it only limits agent runs actively executing at a given moment.
  • The semaphore is sized once when the orchestrator is constructed at server startup. Changing CREWSHIP_MAX_CONCURRENT_RUNS or orchestrator.max_concurrent_runs requires a server restart to take effect.
  • There is no hard upper limit, but a value above 128 is treated as suspicious: the server emits a loud WARN at startup (orchestrator max concurrent runs is far above the default) so an accidental extra zero — an intended 80 typed as 800, or a stray 1000000 — is caught before it saturates the daemon. The value is still honored; the warning is advisory only.

Task States

SKIPPED tasks are treated as terminal alongside COMPLETED and FAILED when checking mission completion. A skipped task does not block downstream dependencies and does not cause mission failure.

Token Budget Calculation

The orchestrator allocates system prompt space using a token budget system defined in internal/tokenutil: The allocation works as follows:
After conversation and memory injection, the orchestrator appends additional context blocks in order: lead crew context (for LEAD agents), peer communication context (for crew AGENT members).

Mission Brief Construction

When an agent is dispatched for a mission task, the buildMissionBrief function constructs a rich context prompt with five sections. The total brief is capped at 32KB (maxBriefTotalLen); if exceeded, the brief is truncated with a note.

1. IMPORTANT Preamble

Only included when dependency outputs exist. Instructs the agent not to ask clarifying questions:

2. [MISSION]

Mission title, goal, and a DAG overview listing all tasks with their status markers:
  • + COMPLETED
  • > IN_PROGRESS
  • x FAILED
  • PENDING/BLOCKED

3. [INPUT FROM PREVIOUS TASKS]

Outputs from completed dependency tasks, injected before the assignment so agents read context first. When a task produced a structured handoff block, only the handoff summary, artifacts, and confidence are included (more concise). Otherwise the full result summary is included, truncated to 4,000 characters per dependency.

4. [YOUR ASSIGNMENT]

The specific task title, description, and iteration number (if this is a retry).

5. [OUTPUT FORMAT]

Structured handoff instructions requiring the agent to produce a ---HANDOFF--- block with summary, confidence, and artifacts.

Lead Planning Phase

When a mission starts with 0 tasks, the engine dispatches the Lead agent to create a plan. The Lead uses its crew context to understand available agents and creates tasks via the sidecar /mission/create endpoint.

LeadPlanning Flag

The DispatchRequest includes a LeadPlanning flag that tells the API layer to dispatch the agent as a LEAD with sidecar access. This is essential because Lead agents need access to the mission management API (/mission/create, /mission/{id}) to define tasks, while regular AGENT tasks skip the sidecar for security.

TOCTOU Prevention

A time-of-check-to-time-of-use race is prevented by inserting a sentinel missionState into the active map before loading the mission from the database. The planningDispatched flag on the mission state prevents re-dispatching the Lead if it is still working. This flag is only set to true after dispatchLeadPlanning succeeds.

Scaling Rules

The Lead agent follows complexity-based scaling rules injected via the system prompt:

Workflow Templates

Four built-in workflow templates are defined in internal/orchestrator/workflow.go:
Tasks execute one after another in order.

The Ralph Loop Pattern

The LoopController (internal/orchestrator/loop.go) manages task retry logic:
  1. When a task fails and has max_iterations > 1, the controller increments the iteration counter and resets the task to PENDING
  2. For loop-back patterns (dev-test-loop), when a downstream task fails, the upstream task is reset to restart the cycle
  3. Previous failure context from the progress log is injected so the agent learns from mistakes
The ShouldRetry method checks if a failed task has remaining iterations. If yes, it resets the task:
  • Status back to PENDING
  • Iteration counter incremented
  • All execution fields cleared (assignment_id, result_summary, error_message, started_at, completed_at, duration_ms)
The RetryLoopBack method handles the upstream reset pattern: when a downstream task (e.g., “test”) fails, it checks the dependency chain. If an upstream task (e.g., “develop”) has remaining iterations, that task is reset to PENDING and the failed downstream task is set to BLOCKED — ready to run again once the upstream completes.
Tasks without max_iterations set (or max_iterations <= 1) are never retried. A failed task without retry configuration causes the mission to fail.

Task Approval Gate

The checkApprovalGate function determines whether a completed task should be held for human review. The gate evaluates three inputs:
  1. Explicit flag — if approval_required = 1 on the task, it is always held
  2. Confidence threshold — the agent’s self-reported confidence from the handoff block
  3. Escalation config — per-crew configuration with tiered thresholds

Escalation Config

Each crew can define an escalation_config JSON object with three thresholds:
The evaluation order is:
  1. If confidence >= auto_approve_threshold, return COMPLETED
  2. If approval_required is explicitly set, return AWAITING_APPROVAL
  3. If no config or no confidence data, return COMPLETED
  4. If confidence < require_approval_below, return AWAITING_APPROVAL
  5. If confidence < notify_threshold, send notification but return COMPLETED

Approving or Rejecting Tasks

The ApproveTask method transitions a task from AWAITING_APPROVAL:
  • Approved: task moves to COMPLETED, dependent BLOCKED tasks are unblocked
  • Rejected: task moves to FAILED, all downstream dependent tasks are recursively failed with reason “upstream task rejected”
Approval requires a userID for the audit trail. The approval status (APPROVED or REJECTED), approver, timestamp, and evaluation notes are persisted on the task.
When a task is held in AWAITING_APPROVAL, the mission engine sends an approval.required WebSocket message to the workspace so dashboards can display a badge or notification.

Circular Dependency Detection

The ValidateDAG method checks all mission tasks for:
  1. References to nonexistent task IDs — any depends_on entry that does not match an existing task ID causes validation to fail
  2. Circular dependencies — detected using Kahn’s algorithm (topological sort)
DAG validation runs before the mission loop begins scheduling, preventing tasks from being dispatched into an unresolvable dependency graph. The error message reports the number of tasks involved in the cycle: "circular dependency detected: N tasks involved in cycle".
The implementation builds an adjacency list and computes in-degrees for each task:

Deadlock Detection

The mission engine detects deadlocks when all remaining tasks are BLOCKED with no task currently IN_PROGRESS, PENDING, or AWAITING_APPROVAL. The detection logic:
  1. If any task is PENDING, IN_PROGRESS, or AWAITING_APPROVAL — not deadlocked (progress is still possible)
  2. COMPLETED, SKIPPED, and FAILED tasks are terminal — they cannot contribute to progress
  3. If all non-terminal tasks are BLOCKED — deadlock confirmed
When a deadlock is detected:
  1. The mission is marked as FAILED
  2. A mission_deadlock progress event is emitted
  3. All AWAITING_APPROVAL tasks are failed with “mission timed out”

Circuit Breaker

The circuit breaker tracks consecutive failures per agent. After 3 consecutive failures (circuitBreakerThreshold), the agent is considered unhealthy and tasks are not dispatched to it.

CooldownManager

The CooldownManager (internal/orchestrator/failover.go) handles rate limit detection and credential cooldown. When an agent run fails due to a rate limit, the associated credential is placed in a cooldown period to avoid hammering the provider.

Rate Limit Detection

The IsRateLimitError function checks stderr output against known patterns. Detection requires exit code 1 and a case-insensitive match against any of these patterns.

Cooldown Behavior

When a rate limit is detected:
  1. MarkCooldown(credentialID, 5*time.Minute) places the credential in a 5-minute cooldown
  2. IsInCooldown(credentialID) returns true during this period, causing the orchestrator to skip that credential
  3. ClearExpired() removes stale entries
The cooldown is per-credential, not per-agent. If an agent has multiple credentials assigned, only the rate-limited credential is paused — the orchestrator can fall back to an alternate credential.

Progress Logging

The ProgressWriter (internal/orchestrator/progress.go) appends structured JSONL events to a per-mission progress file at data/crews/{crewSlug}/missions/{traceID}/progress.jsonl.

Event Types

Each event includes a UTC timestamp.
The progress file is append-only and agents can read it during retry iterations to understand what happened in previous attempts (the Ralph Loop “external state” pattern). The BuildProgressContext method formats the JSONL into a human-readable text block suitable for injection into an agent’s system prompt.

Structured Handoff

Agents produce structured handoff data at the end of tasks:
The parseHandoff function extracts this structure from agent output. Both summary and confidence are required for a valid handoff — partial blocks are treated as unparsed. The confidence value (low, medium, high) feeds into the approval gate. When parsed as a float (via escalation config), it determines whether the task auto-approves or requires human review.

Cross-Crew Work

Crews are isolated by default. A crew link (Settings → Crew links, or crewship crew connect) is what lets one hand work to another; it is created by a workspace admin and enforced server-side on every dispatch, message and shared-file access. A lead assigns across a link by naming the crew:
Omit crew for your own crew. Results come back from /results/{assignment_id} either way, and curl -s http://localhost:9119/connections lists the links as they stand right now. A lead’s system prompt already names the crews it can reach and who is in them, so it does not have to discover them. Peer queries (/query) are crew-local by design and cannot cross a link. Mission tasks can likewise reference agents in linked crews; the system routes each assignment to the right crew container.
A one-way link points one way. If Ops → Engineering is unidirectional, Ops may dispatch into Engineering and not the reverse — the reverse attempt is refused with “crews are not connected”.
Crew-to-crew handoff with critique exchange (e.g. backend crew hands a draft to a testing crew for review) is on the v0.2 roadmap.

Delegation limits

Any agent in a crew can hand work to a crew member with /assign — leads and crew members alike. What bounds it is not who you are, it is how deep the chain already is and how much that run already has out. Both are instance settings, read live — a change applies to the next dispatch, not the next restart:
Past either limit, /assign answers 403 with a message that names the limit and the setting, and no assignment row is written. The agent sees it and is told (in its system prompt) to do the work itself or report back rather than retry.
The numbers multiply. One originating run at the defaults bounds a tree at 8 + 64 dispatches, not 16. Raise max_depth with that in mind — and note that actual concurrency stays far lower, because each crew’s slot budget (max_concurrent_agents, or memory ÷ runtime.agent_min_memory_mb) queues everything past it.
How the depth is known. It is not sent by the agent, and it cannot be. The sidecar resolves who is calling from that agent’s own bearer token, and crewshipd reads the depth off the assignment row that agent is currently executing (assignments.depth, with parent_assignment_id recording the chain). A depth field in the request body is read by nothing.
/query’s own depth >= 2 check is a different, weaker thing: it reads the depth from the request body. Do not model new controls on it.

Mission limits

The delegation limits above bound /assign. They do not bound a mission: the mission engine dispatches its task list through its own path, so a mission an agent authored is not a delegation hop and is not counted as one. Missions are bounded by their own two numbers, on the agent door (POST /mission/create): Both are instance settings, read live — a change applies to the next call:
Past either limit, /mission/create answers 403 with a body naming the setting an operator would change, and no mission or task row is written.
The second number is the one that bounds recursion. A mission created with no tasks makes the engine run its lead as a planning turn, and a planning turn can create another mission — so a per-mission task cap on its own would bound nothing. The crew’s live-mission budget does, because every mission an agent creates lands in the crew its token is bound to.
Missions you create through the dashboard or the JWT API are not capped and are not counted against the agents’ budget: an operator planning work is making a decision, not fanning out unattended.

Sidecar API for Orchestration

Any crew agent can reach the sidecar proxy at localhost:9119; the orchestration routes are:

Sidecar API for the issue board

Agents are participants on the issue board, not just reporters. Every verb below runs on the same sidecar proxy and is authenticated the same way (bearer token on file descriptor 3 — never on a command line), the GETs included. Identifiers are the human-readable ones (ENG-42), not row ids.

Updating an issue

Send only the fields you are changing — a PATCH carrying just a status does not blank the assignee. Status changes must follow the board’s workflow; an illegal jump comes back 400 naming the transition it refused.

Splitting an issue into sub-issues

Decomposition is the reason the link verb exists: a large issue is broken into pieces, each piece becomes a child issue, and each child can then be assigned its own agent.
  1. POST /issue/create once per piece.
  2. POST /issue/{child}/link with {"target_identifier":"<parent>","relation_type":"sub_issue_of"}.
  3. PATCH /issue/{child} with the assignee for that piece.
relation_type is one of blocks, blocked_by, relates_to, duplicate_of or sub_issue_of. blocked_by is stored as the inverse blocks link, so the board never holds two shapes of the same relationship.

What an agent cannot do here

These are enforced on the server, not by convention — an agent that tries gets a 403 or a 400, and retrying will not help:
  • Author as someone else. A crew shares one container and one sidecar, so an agent_id in the request body is ignored; the author is always the agent whose bearer token made the call.
  • Change another crew’s issues. The workspace and crew come from the sidecar’s own token binding, not from the request, and every verb is gated on the issue named in the path. The single exception is the link target: a relation may point at another crew’s issue in the same workspace, because it does not modify that issue. sub_issue_of writes parent_issue_id on the path issue, so it is gated like any other mutation.
  • Rewrite the issue itself. title and description are not updatable through these verbs, and neither are project_id, milestone_id, routine_id or sort_order. Agents report progress; they do not rewrite the brief a human wrote.
  • Delete anything. There is no delete verb on this surface.
Issue titles, descriptions and comments an agent reads back arrive wrapped in an <untrusted …> block. Anyone who can file an issue chooses those bytes, so they are handed to the model as data rather than as instructions — the same fence the orchestrator uses for webhook payloads and mission comments. See Lookout.

What’s Next