Orchestration
Crewship’s orchestration system manages multi-agent missions through theMissionEngine (internal/orchestrator/mission.go). It handles task scheduling, dependency resolution, failure recovery, and cross-crew coordination.
“Orchestration” here means the engine subsystem in
internal/orchestrator/, not a navigable page. After the Plan/Run/Build/System IA refactor, the user-facing surfaces are split: Routines for reusable recipes, Issues for the work-item tracker, Inbox for your actionable feed, and Activity for the live trace canvas. The legacy /orchestration route now soft-redirects to /activity.Mission Lifecycle
- PLANNING — Mission created, tasks defined (or waiting for Lead to plan)
- IN_PROGRESS — Tasks being scheduled and executed
- REVIEW — All tasks finished (none failed). The mission enters review before final completion, allowing humans to inspect results
- COMPLETED — Mission accepted after review
- FAILED — A task failed and could not recover, or deadlock/timeout detected
- CANCELLED — Manually stopped by user or system
The REVIEW state is inserted between IN_PROGRESS and COMPLETED. When all tasks reach a terminal state (COMPLETED, FAILED, or SKIPPED) and none have failed, the mission transitions to REVIEW rather than directly to COMPLETED. If any task failed, the mission transitions to FAILED instead.
The Mission Engine
TheMissionEngine is the central orchestrator. Its behaviour is governed by a handful of fixed parameters:
Mission Loop
TherunMissionLoop function runs as a goroutine for each active mission. Every 3 seconds it:
Restart Durability
Mission loops are in-memory goroutines, but missions survive server restarts. At boot — right after orphaned-run recovery — the server scans the database for missions still inIN_PROGRESS and re-attaches an orchestration loop to each one. Tasks the previous process never dispatched are picked up on the next tick, and BLOCKED tasks whose dependencies already completed are self-healed back to PENDING. No operator action is needed after a restart or crash; the 2-hour mission timeout restarts from the moment of re-attach.
Scaling / Concurrency
Separate from per-mission task scheduling above, the orchestrator bounds how many agent-run exec fan-outs (RunAgent) may be in flight across the whole server at once, via a semaphore (runSem in internal/orchestrator/orchestrator.go). Each run fans out roughly a dozen container execs — sidecar start, mkdir/manifest/credential setup, and the long-lived agent CLI exec — all against a single Docker daemon, so unbounded concurrency here is the fastest path to daemon saturation or host OOM.
Precedence when both are set:
CREWSHIP_MAX_CONCURRENT_RUNS (env) wins over orchestrator.max_concurrent_runs (config file), which wins over the built-in default of 8.
- Raising the cap increases parallelism, but the practical ceiling is whichever comes first: host RAM (each run’s container + agent CLI + MCP processes) or your LLM provider’s own rate limits.
- Idle agents and provisioned-but-inactive crews cost nothing against this cap — it only limits agent runs actively executing at a given moment.
- The semaphore is sized once when the orchestrator is constructed at server startup. Changing
CREWSHIP_MAX_CONCURRENT_RUNSororchestrator.max_concurrent_runsrequires a server restart to take effect. - There is no hard upper limit, but a value above 128 is treated as suspicious: the server emits a loud
WARNat startup (orchestrator max concurrent runs is far above the default) so an accidental extra zero — an intended80typed as800, or a stray1000000— is caught before it saturates the daemon. The value is still honored; the warning is advisory only.
Task States
SKIPPED tasks are treated as terminal alongside COMPLETED and FAILED when checking mission completion. A skipped task does not block downstream dependencies and does not cause mission failure.
Token Budget Calculation
The orchestrator allocates system prompt space using a token budget system defined ininternal/tokenutil:
The allocation works as follows:
Mission Brief Construction
When an agent is dispatched for a mission task, thebuildMissionBrief function constructs a rich context prompt with five sections. The total brief is capped at 32KB (maxBriefTotalLen); if exceeded, the brief is truncated with a note.
1. IMPORTANT Preamble
Only included when dependency outputs exist. Instructs the agent not to ask clarifying questions:2. [MISSION]
Mission title, goal, and a DAG overview listing all tasks with their status markers:+COMPLETED>IN_PROGRESSxFAILEDPENDING/BLOCKED
3. [INPUT FROM PREVIOUS TASKS]
Outputs from completed dependency tasks, injected before the assignment so agents read context first. When a task produced a structured handoff block, only the handoff summary, artifacts, and confidence are included (more concise). Otherwise the full result summary is included, truncated to 4,000 characters per dependency.4. [YOUR ASSIGNMENT]
The specific task title, description, and iteration number (if this is a retry).5. [OUTPUT FORMAT]
Structured handoff instructions requiring the agent to produce a---HANDOFF--- block with summary, confidence, and artifacts.
Lead Planning Phase
When a mission starts with 0 tasks, the engine dispatches the Lead agent to create a plan. The Lead uses its crew context to understand available agents and creates tasks via the sidecar/mission/create endpoint.
LeadPlanning Flag
TheDispatchRequest includes a LeadPlanning flag that tells the API layer to dispatch the agent as a LEAD with sidecar access. This is essential because Lead agents need access to the mission management API (/mission/create, /mission/{id}) to define tasks, while regular AGENT tasks skip the sidecar for security.
TOCTOU Prevention
A time-of-check-to-time-of-use race is prevented by inserting a sentinelmissionState into the active map before loading the mission from the database. The planningDispatched flag on the mission state prevents re-dispatching the Lead if it is still working. This flag is only set to true after dispatchLeadPlanning succeeds.
Scaling Rules
The Lead agent follows complexity-based scaling rules injected via the system prompt:Workflow Templates
Four built-in workflow templates are defined ininternal/orchestrator/workflow.go:
- Sequential
- Parallel
- Dev-Test Loop
- Pipeline
Tasks execute one after another in order.
The Ralph Loop Pattern
TheLoopController (internal/orchestrator/loop.go) manages task retry logic:
- When a task fails and has
max_iterations > 1, the controller increments the iteration counter and resets the task toPENDING - For loop-back patterns (dev-test-loop), when a downstream task fails, the upstream task is reset to restart the cycle
- Previous failure context from the progress log is injected so the agent learns from mistakes
ShouldRetry method checks if a failed task has remaining iterations. If yes, it resets the task:
- Status back to
PENDING - Iteration counter incremented
- All execution fields cleared (
assignment_id,result_summary,error_message,started_at,completed_at,duration_ms)
RetryLoopBack method handles the upstream reset pattern: when a downstream task (e.g., “test”) fails, it checks the dependency chain. If an upstream task (e.g., “develop”) has remaining iterations, that task is reset to PENDING and the failed downstream task is set to BLOCKED — ready to run again once the upstream completes.
Task Approval Gate
ThecheckApprovalGate function determines whether a completed task should be held for human review. The gate evaluates three inputs:
- Explicit flag — if
approval_required = 1on the task, it is always held - Confidence threshold — the agent’s self-reported confidence from the handoff block
- Escalation config — per-crew configuration with tiered thresholds
Escalation Config
Each crew can define anescalation_config JSON object with three thresholds:
The evaluation order is:
- If confidence >=
auto_approve_threshold, return COMPLETED - If
approval_requiredis explicitly set, return AWAITING_APPROVAL - If no config or no confidence data, return COMPLETED
- If confidence <
require_approval_below, return AWAITING_APPROVAL - If confidence <
notify_threshold, send notification but return COMPLETED
Approving or Rejecting Tasks
TheApproveTask method transitions a task from AWAITING_APPROVAL:
- Approved: task moves to COMPLETED, dependent BLOCKED tasks are unblocked
- Rejected: task moves to FAILED, all downstream dependent tasks are recursively failed with reason “upstream task rejected”
userID for the audit trail. The approval status (APPROVED or REJECTED), approver, timestamp, and evaluation notes are persisted on the task.
When a task is held in AWAITING_APPROVAL, the mission engine sends an
approval.required WebSocket message to the workspace so dashboards can display a badge or notification.Circular Dependency Detection
TheValidateDAG method checks all mission tasks for:
- References to nonexistent task IDs — any
depends_onentry that does not match an existing task ID causes validation to fail - Circular dependencies — detected using Kahn’s algorithm (topological sort)
"circular dependency detected: N tasks involved in cycle".
Kahn's Algorithm internals
Kahn's Algorithm internals
The implementation builds an adjacency list and computes in-degrees for each task:
Deadlock Detection
The mission engine detects deadlocks when all remaining tasks areBLOCKED with no task currently IN_PROGRESS, PENDING, or AWAITING_APPROVAL. The detection logic:
- If any task is PENDING, IN_PROGRESS, or AWAITING_APPROVAL — not deadlocked (progress is still possible)
- COMPLETED, SKIPPED, and FAILED tasks are terminal — they cannot contribute to progress
- If all non-terminal tasks are BLOCKED — deadlock confirmed
- The mission is marked as
FAILED - A
mission_deadlockprogress event is emitted - All
AWAITING_APPROVALtasks are failed with “mission timed out”
Circuit Breaker
The circuit breaker tracks consecutive failures per agent. After 3 consecutive failures (circuitBreakerThreshold), the agent is considered unhealthy and tasks are not dispatched to it.
CooldownManager
TheCooldownManager (internal/orchestrator/failover.go) handles rate limit detection and credential cooldown. When an agent run fails due to a rate limit, the associated credential is placed in a cooldown period to avoid hammering the provider.
Rate Limit Detection
TheIsRateLimitError function checks stderr output against known patterns. Detection requires exit code 1 and a case-insensitive match against any of these patterns.
Rate-limit patterns
Rate-limit patterns
Cooldown Behavior
When a rate limit is detected:MarkCooldown(credentialID, 5*time.Minute)places the credential in a 5-minute cooldownIsInCooldown(credentialID)returns true during this period, causing the orchestrator to skip that credentialClearExpired()removes stale entries
Progress Logging
TheProgressWriter (internal/orchestrator/progress.go) appends structured JSONL events to a per-mission progress file at data/crews/{crewSlug}/missions/{traceID}/progress.jsonl.
Event Types
Each event includes a UTC timestamp.Progress event reference
Progress event reference
BuildProgressContext method formats the JSONL into a human-readable text block suitable for injection into an agent’s system prompt.
Structured Handoff
Agents produce structured handoff data at the end of tasks:parseHandoff function extracts this structure from agent output. Both summary and confidence are required for a valid handoff — partial blocks are treated as unparsed.
The confidence value (low, medium, high) feeds into the approval gate. When parsed as a float (via escalation config), it determines whether the task auto-approves or requires human review.
Cross-Crew Work
Crews are isolated by default. A crew link (Settings → Crew links, orcrewship crew connect) is
what lets one hand work to another; it is created by a workspace admin and enforced server-side on
every dispatch, message and shared-file access.
A lead assigns across a link by naming the crew:
crew for your own crew. Results come back from /results/{assignment_id} either way, and
curl -s http://localhost:9119/connections lists the links as they stand right now.
A lead’s system prompt already names the crews it can reach and who is in them, so it does not have to
discover them. Peer queries (/query) are crew-local by design and cannot cross a link.
Mission tasks can likewise reference agents in linked crews; the system routes each assignment to the
right crew container.
Crew-to-crew handoff with critique exchange (e.g. backend crew hands a draft to a testing crew for review) is on the v0.2 roadmap.
Delegation limits
Any agent in a crew can hand work to a crew member with/assign — leads and
crew members alike. What bounds it is not who you are, it is how deep the
chain already is and how much that run already has out.
Both are instance settings, read live — a change applies to the next dispatch,
not the next restart:
/assign answers 403 with a message that names the
limit and the setting, and no assignment row is written. The agent sees it and
is told (in its system prompt) to do the work itself or report back rather than
retry.
The numbers multiply. One originating run at the defaults bounds a tree at
8 + 64 dispatches, not 16. Raise
max_depth with that in mind — and note
that actual concurrency stays far lower, because each crew’s slot budget
(max_concurrent_agents, or memory ÷ runtime.agent_min_memory_mb) queues
everything past it.assignments.depth, with parent_assignment_id recording the
chain). A depth field in the request body is read by nothing.
Mission limits
The delegation limits above bound/assign. They do not bound a mission:
the mission engine dispatches its task list through its own path, so a mission
an agent authored is not a delegation hop and is not counted as one. Missions
are bounded by their own two numbers, on the agent door
(POST /mission/create):
Both are instance settings, read live — a change applies to the next call:
/mission/create answers 403 with a body naming the
setting an operator would change, and no mission or task row is written.
The second number is the one that bounds recursion. A mission created with no
tasks makes the engine run its lead as a planning turn, and a planning turn
can create another mission — so a per-mission task cap on its own would bound
nothing. The crew’s live-mission budget does, because every mission an agent
creates lands in the crew its token is bound to.
Sidecar API for Orchestration
Any crew agent can reach the sidecar proxy atlocalhost:9119; the orchestration
routes are:
Sidecar API for the issue board
Agents are participants on the issue board, not just reporters. Every verb below runs on the same sidecar proxy and is authenticated the same way (bearer token on file descriptor 3 — never on a command line), theGETs included.
Identifiers are the human-readable ones (
ENG-42), not row ids.
Updating an issue
Send only the fields you are changing — aPATCH carrying just a status does not
blank the assignee. Status changes must follow the board’s workflow; an illegal
jump comes back 400 naming the transition it refused.
Splitting an issue into sub-issues
Decomposition is the reason the link verb exists: a large issue is broken into pieces, each piece becomes a child issue, and each child can then be assigned its own agent.POST /issue/createonce per piece.POST /issue/{child}/linkwith{"target_identifier":"<parent>","relation_type":"sub_issue_of"}.PATCH /issue/{child}with the assignee for that piece.
relation_type is one of blocks, blocked_by, relates_to, duplicate_of or
sub_issue_of. blocked_by is stored as the inverse blocks link, so the board
never holds two shapes of the same relationship.
What an agent cannot do here
These are enforced on the server, not by convention — an agent that tries gets a403 or a 400, and retrying will not help:
- Author as someone else. A crew shares one container and one sidecar, so an
agent_idin the request body is ignored; the author is always the agent whose bearer token made the call. - Change another crew’s issues. The workspace and crew come from the sidecar’s own
token binding, not from the request, and every verb is gated on the issue named in the
path. The single exception is the link target: a relation may point at another crew’s
issue in the same workspace, because it does not modify that issue.
sub_issue_ofwritesparent_issue_idon the path issue, so it is gated like any other mutation. - Rewrite the issue itself.
titleanddescriptionare not updatable through these verbs, and neither areproject_id,milestone_id,routine_idorsort_order. Agents report progress; they do not rewrite the brief a human wrote. - Delete anything. There is no delete verb on this surface.
Issue titles, descriptions and comments an agent reads back arrive wrapped in an
<untrusted …> block. Anyone who can file an issue chooses those bytes, so they
are handed to the model as data rather than as instructions — the same fence the
orchestrator uses for webhook payloads and mission comments. See
Lookout.What’s Next
- Agent memory — persistent agent memory across sessions
- Scheduling — cron-based automated agent runs