Routines

​

What is a Routine?

• A JSON DSL definition — inputs, outputs, ordered or DAG-structured steps, validation gates, declared egress, credential requirements
• Authorship metadata — which crew, which agent (or user), via which path
• Triggers — cron schedules and webhook tokens that fire it autonomously
• Run history — every invocation as an immutable journal trace with step-level events
• Versions — publishing creates an immutable version; rollback is one API call

​

The routines page

​

The detail page

​

Edit and Test

• Save draft saves work without publishing or running it. A conflicting edit reports the conflict and preserves local work.
• Test routine checks the recipe, references and static prerequisites. It does not run agents, scripts or HTTP calls and cannot prove a real run will succeed.
• Test this step uses sample data. Transforms calculate a result; agent, HTTP and script steps require an explicit replacement output and make no external calls. Unsupported step types cannot execute through this test.
• Publish performs the required recipe test and publishes the draft after applicable risk review. It is one explicit action, separate from saving a draft.
• Run starts actual work. It can change data in connected systems and incur costs. With inputs, it first presents their typed form. A new run is recorded in History; an existing run retains its accepted snapshot after later edits.

​

Three authoring paths

​

Authoring with AI (one-shot)

crewship routine capabilities acct              # readable summary
crewship routine capabilities acct -f json      # full machine-readable bundle
crewship routine capabilities acct -f json | claude -p "write a routine that reconciles this month's invoices"

• schema — the routine DSL JSON schema (the same bytes as routine schema), nested inline so the whole contract travels in one response.
• container — the crew’s resolved devcontainer capabilities: datastores (host/port to declare under resources.datastores) and installed CLI tools.
• integrations — connected integrations WITH their enabled tool names (GMAIL_FETCH_EMAIL, …), so the author references real tools, not guesses.
• agents — the crew’s agent slugs, which agent_run steps reference.
• runtimes — the truth about what’s wired in this build: code.wired (expr, cel) vs code.reserved_unwired (python/go/bash — legal names but no runner, so don’t use them), and the type: script interpreter inference table (.py → python3, …).

​

DSL anatomy

{
  "dsl_version": "1.0",
  "name": "summarize-text",
  "description": "Summarize input text in 3 bullets.",
  "inputs": [
    { "name": "text", "type": "string", "required": true }
  ],
  "outputs": [
    { "name": "summary", "type": "string" }
  ],
  "steps": [
    {
      "id": "summarize",
      "type": "agent_run",
      "agent_slug": "tomas",
      "complexity": "fast",
      "prompt": "Summarize: {{ inputs.text }}"
    }
  ]
}

​

YAML instead of JSON

dsl_version: "1.0"
name: summarize-text
# comments are fine — YAML only, dropped on conversion, same as they
# always were once JSON.Marshal'd a hand-authored JSON file
description: Summarize input text in 3 bullets.
inputs:
  - name: text
    type: string
    required: true
outputs:
  - name: summary
    type: string
steps:
  - id: summarize
    type: agent_run
    agent_slug: tomas
    complexity: fast
    prompt: |
      Summarize the following text in exactly 3 bullets:
      {{ inputs.text }}

​

Top-level fields

​

InputSpec

{
  "name": "since",
  "type": "string",
  "required": false,
  "default": "yesterday",
  "description": "ISO date or 'yesterday'/'today'",
  "min": 0,
  "max": 100
}

​

Template substitution

• inputs.X — declared input value
• steps.Y.output — full text output of an earlier step
• steps.Y.output.path — JSON path into a step’s output (when output parses as JSON)
• env.AUTHOR_CREW_NAME / etc. — read-only allowlist of execution context
• secrets.<type> — a workspace-vault credential, resolved at run time by type (see Secrets below)
• routine.state.<key> — durable cross-run state carried forward from a previous run (see Cross-run state below)

​

Secrets ({{ secrets.<type> }})

{
  "id": "charge",
  "type": "script",
  "script": {
    "path": "scripts/charge.py",
    "env": { "STRIPE_KEY": "{{ secrets.stripe }}" }
  }
}

​

Cross-run state ({{ routine.state.<key> }})

• Read with {{ routine.state.<key> }}. A key that was never written renders empty (like a missing input). The value you read is the snapshot as of run start — it reflects what a prior run wrote, not a write earlier in this same run.
• Write with a step’s state_write binding — a map of key → template, rendered after the step completes (so it can reference the step’s own output) and persisted for the next run:

{
  "id": "sync",
  "type": "agent_run",
  "agent_slug": "importer",
  "prompt": "Import records created after id {{ routine.state.last_id }}. Return the highest id you imported.",
  "state_write": { "last_id": "{{ steps.sync.output }}" }
}

​

Inspecting and repairing state

# Which cursor is stuck, and when was it last written?
crewship routine state list nightly-import

# Rewind one schedule's watermark to a known-good value
crewship routine state set nightly-import last_id 41200 --schedule psched_abc123

# Drop one key, or the whole bucket
crewship routine state rm    nightly-import last_id --schedule psched_abc123
crewship routine state clear nightly-import          --schedule psched_abc123

​

Required integrations

{
  "name": "triage-incident",
  "integrations_required": ["slack", "github"],
  "steps": [ ... ]
}

• Declared is always allowed. Saving a routine that names an integration the crew lacks is fine — declaring is a contract, not a connection. Only the run enforces. (Save-time validation only checks the list is well-formed: non-empty slugs, lowercased/trimmed, within a sane count cap.)
• Enforced at run time. Before a run starts, Crewship resolves the integrations the routine’s author crew has connected and compares them to integrations_required. If any are missing the run is blocked with an RFC 7807 Problem Details response, HTTP 422, carrying a machine-readable missing_integrations: string[] member and a human detail like routine requires integration "slack" not connected for crew "Marketing". The UI uses missing_integrations to render a Connect action. The run never starts — no tokens spent.
• No-op fast path. An empty / absent integrations_required does zero resolution work — no overhead for routines that don’t use it.
• Fail-open. If integration-availability resolution itself errors, the run is allowed (a warning is logged). A bug in resolution must never wedge every run of every routine — a forgotten integration is a soft, recoverable failure; a hard block on all runs would be a self-inflicted outage.
• run is gated; dry_run is not. A live run executes against the author crew’s agents, so it’s gated (fail fast rather than land an unrunnable routine). The internal save gate’s draft validation applies the same integration check. dry_run is a preview that invokes nothing, so it’s left ungated — it shows what the routine would need, even integrations the crew hasn’t connected yet.

​

Required credentials

{
  "name": "charge-and-notify",
  "credentials_required": [{ "type": "stripe" }],
  "steps": [ ... ]
}

• Declared is always allowed. Saving/importing a routine that names a credential the vault lacks is fine — you can connect it later. Save-time validation only checks the list is well-formed (non-empty type, deduped, within a count cap).
• Enforced at run time. Before a live run starts, Crewship probes the author crew’s workspace vault for an ACTIVE credential of each declared type. If any are missing the run is blocked with an RFC 7807 Problem Details response, HTTP 422, carrying a machine-readable missing_credentials: string[] member and a human detail. No tokens spent, no half-run with an opaque auth failure deep in a step.
• No-op fast path. An empty / absent credentials_required does zero probe work.
• Unavailable is not available. A credential probe error (including an unavailable database) blocks dispatch with HTTP 503. Retry after the store recovers. A confirmed missing credential remains HTTP 422. In-process child calls and scheduled dispatch use the same blocking gate.
• Enforced wherever integrations_required is. Live run, run_batch, internal runs, and the test_run save-preview gate all check it — the same precondition boundary that already probes for required integrations and resources. Only the executor’s pure dry_run mode — which invokes no steps and resolves no secrets — is exempt. Persisting a well-formed draft is still fine (see above); it’s running and the test-run preview that block until the credential is connected.

​

Capability manifest

​

The resources block

{
  "name": "deploy-service",
  "resources": {
    "datastores": [
      { "type": "postgres", "name": "main", "note": "writes table runs" },
      { "type": "redis",    "name": "cache" }
    ],
    "tools": [
      { "type": "ansible",  "name": "deploy.yml" },
      { "type": "kubectl" }
    ]
  },
  "steps": [ ... ]
}

• datastores[].type — any string, but use the canonical vocabulary redis | postgres | mysql | mongodb | other so it matches what the crew’s container catalog reports (the precondition gate compares type case-insensitively; a non-canonical type simply won’t match a crew resource).
• tools[].type — any string; canonical examples ansible | terraform | kubectl | bash | python | other. Same matching rule as datastores.
• name / note are free-form and optional.

​

Required resources

• Enforced at run time. Before a run starts, Crewship resolves the author crew’s container resources and compares them to the declared resources.datastores + resources.tools. If the routine requires a datastore or tool the crew doesn’t have, the run is blocked with an RFC 7807 Problem Details response, HTTP 422, carrying a machine-readable missing_resources: [{ kind, type, name }] member (kind is "datastore" or "tool") and a human detail like routine needs datastore postgres, tool ansible, not available to crew "Ops". The run never starts — no tokens spent — until the resource is provisioned on the crew.
• Matching. A required datastore is satisfied when the crew has a datastore of the same engine type (case-insensitive; the service name is advisory and not matched). A required tool is satisfied when the crew has an installed tool whose name matches the required type (the tool’s name, e.g. deploy.yml, is the concrete artifact and isn’t matched against the container).
• No-op fast path. A routine that declares no resources block does zero resolution work. Only the declared resources are gated — code-step runtimes folded into the manifest’s tools (e.g. cel) are internal executor runtimes, not crew CLIs, and are never gated.
• Fail-open. If the routine has no author crew, or resource resolution itself errors, the run is allowed (a warning is logged) — same reasoning as the integration gate: a resolver bug must never wedge every run.
• run is gated; dry_run is not. (The internal save gate’s draft validation applies the same resource check.)

​

Agents already know what the container has

​

Step types

​

agent_run

{
  "id": "summarize",
  "type": "agent_run",
  "agent_slug": "tomas",
  "complexity": "fast",
  "prompt": "Summarize: {{ inputs.text }}",
  "model_override": null,
  "timeout_seconds": 60,
  "validation": {
    "schema": { "type": "object", "required": ["title"] },
    "must_not_contain": ["API_KEY=", "Bearer "],
    "min_length": 30
  },
  "on_fail": "escalate_tier"
}

​

Retry transient failures

{
  "id": "call_flaky_api",
  "type": "http",
  "http": { "method": "GET", "url": "https://api.example.com/status" },
  "retry": {
    "max_attempts": 4,
    "backoff": { "min_ms": 1000, "max_ms": 30000, "factor": 2, "jitter": true },
    "retry_on": "transient"
  }
}

​

call_pipeline

{
  "id": "review_summary",
  "type": "call_pipeline",
  "pipeline_slug": "human-approval-step",
  "inputs": {
    "content": "{{ steps.summarize.output }}",
    "approver_role": "marketing_lead"
  }
}

​

Chain depth — the composition budget

• A run a person started is depth 0.
• A routine called by call_pipeline is parent + 1.
• A run fired by an automation reacting to an event is that event’s run + 1.

​

http

{
  "id": "fetch",
  "type": "http",
  "http": {
    "method": "GET",
    "url": "{{ inputs.url }}",
    "max_response_bytes": 200000,
    "success_codes": [200],
    "credential_ref": { "type": "API_KEY", "inject_as": "bearer" }
  },
  "timeout_seconds": 30
}

1. Routine layer — egress_targets. When the routine declares egress_targets, the request host must be one of the declared hosts or a subdomain of one (api.x.com matches target x.com; evilx.com does not). A routine that declares no egress_targets is unrestricted at this layer — backward-compatible with every routine that predates the field. The SSRF guard (private/link-local IPs, DNS-rebind-safe dialing) applies regardless.
2. Crew layer — network policy. The authoring crew’s network policy (network_mode + allowed domains — the same dial that governs the crew’s agent containers) also applies to direct http steps. A restricted crew’s routines can only reach the crew’s allowed domains (exact host match, same as the container proxy). A crew on free mode is not unconditionally open at this layer: an http step that does not declare egress_targets is still held to the same floor restricted mode enforces — the sidecar’s default LLM/CLI provider domains only. This closes an SSRF gap where any create-role member (or a webhook payload driving {{ inputs.url }}) could otherwise point an undeclared http step at an arbitrary public host just because the crew allows its agents free egress. Declaring egress_targets on the routine is the escape hatch — it bypasses this floor and the routine’s own allowlist governs instead (private/link-local IPs still blocked by the SSRF guard regardless). Webhook-triggered runs go further: they’re always held to the restricted floor, regardless of the crew’s own network_mode — the inbound payload is untrusted, so a routine fired by a webhook delivery can’t be used to reach an arbitrary host even if the authoring crew is nominally free. The crew’s own allowed_domains (if configured) still apply on top of the shared defaults, same as a genuinely-restricted crew.

​

wait

{
  "id": "human_approval",
  "type": "wait",
  "wait": {
    "kind": "approval",
    "approval_title": "Approve: {{ inputs.action }}",
    "approval_prompt": "Approve summary?",
    "risk_level": "destructive",
    "timeout_sec": 86400
  }
}

# one run you already have the id for
crewship routine signal run_cmox... --event deploy.finished --payload "v1.4.2"

# every run parked on the event, whoever they are
crewship signal send --event-type deploy.finished --payload "v1.4.2"

​

code

​

runtime: expr (wired, token-zero)

- id: probe
  type: code
  code:
    runtime: expr
    code: "{{ inputs.spend_usd }} > {{ inputs.threshold_usd }}"

• Operators: > >= < <= == !=.
• The body is Render()-ed first, so {{ inputs.x }} / {{ steps.y.output }} placeholders substitute before evaluation.
• Anything that isn’t a single comparison (multiple operators, function calls, arbitrary code) fails closed with a clear error — expr is deliberately not a scripting language.

​

runtime: cel (wired, token-zero, general logic)

- id: spike
  type: code
  code:
    runtime: cel
    # Real logic in one deterministic, token-zero step — no {{ }} needed,
    # inputs are exposed as a typed `inputs` map:
    code: 'inputs.spend_usd > inputs.threshold_usd && inputs.region in ["eu", "us"]'

​

runtime: bash | python | go (rejected at author time)

- id: process
  type: code
  code:
    runtime: python
    code: "import json; print(json.dumps({'sum': sum(input_nums)}))"

​

script

{
  "id": "parse",
  "type": "script",
  "script": {
    "path": "scripts/parse_vypis.py",
    "interpreter": "python3",
    "args": ["{{ inputs.period }}"]
  }
}

{
  "id": "reconcile",
  "type": "agent_run",
  "agent_slug": "accountant",
  "needs": ["parse"],
  "prompt": "Reconcile these parsed movements and flag anomalies:\n{{ steps.parse.output }}"
}

​

Deliver the script into the crew

kind: crew
metadata:
  slug: accounting
spec:
  files:
    - src: scripts/parse_vypis.py        # local path, relative to the manifest
      dest: shared/scripts/parse_vypis.py # lands at /crew/shared/scripts/parse_vypis.py

​

Scaffold and portability

• Scaffold a script-backed routine with crewship routine init --script — it emits a script step feeding a downstream agent_run, with delivery instructions in the description.
• Portable bundles. crewship routine export inlines every referenced script file (base64) from the routine’s author crew, so a portable routine travels with its backbone — recipe + scripts + agent judgment. crewship routine import --crew <slug> re-materializes the inlined scripts into the target crew (same /files/save path). A script whose destination already exists with different content fails loudly — pass --force to overwrite (remember: a crew script is shared across routines, so overwriting may affect another routine). --no-scripts on either command skips inlining/materializing. Even with inlining, the crew manifest remains the source of truth; export/import is a portability convenience.

​

transform

{
  "id": "extract_email",
  "type": "transform",
  "transform": {
    "input": "{{ steps.fetch.output }}",
    "expression": ".items[0].email"
  }
}

​

notify

{
  "id": "tell_client",
  "type": "notify",
  "to": "trigger",
  "title": "Invoices parsed",
  "body": "Parsed {{ steps.extract.output }} invoices.\n\n{{ steps.summarize.output }}",
  "priority": "medium"
}

{
  "id": "tell_ops",
  "type": "notify",
  "to": "workspace",
  "title": "Nightly deploy failed",
  "category": "routines.failed",
  "priority": "high"
}

{ "id": "tell_ops", "type": "notify", "to": "crew:ops", "title": "Nightly reconcile finished" }

​

query

{
  "id": "stats",
  "type": "query",
  "query": { "source": "pipeline_runs", "window_hours": 24 }
}

{
  "id": "summary",
  "type": "transform",
  "needs": ["stats"],
  "transform": { "input": "{{ steps.stats.output }}", "expression": ".summary_md" }
}

​

foreach — fan out over an array

{
  "id": "process_each",
  "type": "foreach",
  "foreach": {
    "items": "{{ inputs.urls }}",
    "as": "url",
    "parallelism": 8,
    "steps": [
      { "id": "fetch",  "type": "http",      "http": { "method": "GET", "url": "{{ inputs.url }}" } },
      { "id": "digest", "type": "agent_run", "agent_slug": "summarizer", "prompt": "Summarize: {{ steps.fetch.output }}" }
    ]
  }
}

​

crewship — act on Crewship’s own nouns

- id: file_it
  type: crewship
  action: issue.create          # namespaced verb
  args:
    title: "Nightly check failed: {{ steps.probe.output }}"
    priority: high

​

Verbs

inputs:
  - name: issue
    type: string          # e.g. ENG-42, passed by the webhook or schedule
steps:
  - id: probe
    type: http
    http: { method: GET, url: "https://status.example.com/api/health" }
  - id: report
    type: crewship
    action: issue.comment
    args:
      identifier: "{{ inputs.issue }}"
      body: |
        Nightly probe says: {{ steps.probe.output }}
  - id: close_it
    type: crewship
    action: issue.update
    if: "{{ steps.probe.output }}"
    args:
      identifier: "{{ inputs.issue }}"
      status: DONE

​

What bounds it

• Autonomy. The verb is gated on the author crew’s autonomy_level. A crew whose level would hold the action for approval gets a refusal instead — a routine has nobody attached to approve it — so on a strict crew, issue writes and assignments do not happen. Raise the level with crewship policy set. The full per-level table, and the reasoning for each cell, is in Autonomy → what a routine may do to the board.
• Escalation volume. escalation.create is allowed at every autonomy level, because an approval to authorise an interrupt is itself the interrupt. What bounds it is a backlog instead: a routine may not escalate while the crew already has escalation.max_pending_per_crew (default 10) unresolved escalations. Resolving them returns the budget.
• Identity is not yours to set. workspace_id, crew_id, agent_id, actor_agent_id, author_agent_id and author_run_id are injected from the run and override anything in args. A routine cannot act as another workspace, a sibling crew, or a different agent — the last one matters because the acting agent is what the delegation depth cap is measured from. A routine with no author agent cannot use issue.comment, which needs an author it can name — so a routine containing that step is refused at save unless one is set. Set it with crewship routine save --author-agent <slug|id> (the agent must belong to --author-crew), or author_agent_id on the save API. The refusal is deliberate: the same routine used to save clean and then fail every run at 03:00 with 400: agent_id is required. The other five verbs have a documented no-agent fallback and are unaffected.
• Chain depth. Whatever the verb creates records the run that created it, so an automation reacting to it inherits this run’s chain budget rather than starting a new one. See Chain depth.
• dry_run does not write. A preview that files a real issue is not a preview; the step reports what it would have called.

​

Conditional if

• CEL expression (no {{ … }}): evaluated against the typed variables inputs, steps, and run. Real comparisons and logic work — "if": "inputs.dry_run == false", "if": "steps.classify == \"spam\"", "if": "inputs.count > 0 && run.is_replay == false". A boolean result gates directly; a number/string result falls through to the truthy rule below.
• Template ("if": "{{ inputs.run_summary }}"): rendered to a plain string, then truthy-checked — empty / false / 0 / no / off (case-insensitive) → step skipped and marked <skipped> in StepOutputs; anything else is truthy. A bare literal like "yes" / "false" follows the same truthy rule.

​

DAG with needs[]

{ "id": "fetch_a", "type": "http", "http": {...} },
{ "id": "fetch_b", "type": "http", "http": {...} },
{ "id": "merge",
  "type": "agent_run",
  "needs": ["fetch_a", "fetch_b"],
  "prompt": "Merge: {{ steps.fetch_a.output }} + {{ steps.fetch_b.output }}"
}

​

parallelism — auto-derive the DAG

{
  "name": "summarize-inbox",
  "parallelism": "auto",
  "steps": [
    { "id": "t1", "type": "agent_run", "agent_slug": "writer", "prompt": "Summarize thread 1" },
    { "id": "t2", "type": "agent_run", "agent_slug": "writer", "prompt": "Summarize thread 2" },
    { "id": "t3", "type": "agent_run", "agent_slug": "writer", "prompt": "Summarize thread 3" }
  ]
}

​

Lifecycle hooks (before_all / after_all / on_failure)

hooks:
  before_all:   # setup — its FAILURE aborts the run before any step executes
    id: claim
    type: http
    http: { method: POST, url: "https://ops.example.com/claim" }
  after_all:    # runs after a COMPLETED run — best-effort
    id: release
    type: http
    http: { method: POST, url: "https://ops.example.com/release" }
  on_failure:   # runs after a FAILED run — best-effort cleanup
    id: alert
    type: http
    http: { method: POST, url: "https://ops.example.com/alert" }

​

Per-step overrides (no version bump)

crewship routine step-override set my-routine summarize \
  --prompt "Summarize in 3 bullets, lead with the risk." --model smart
crewship routine step-override list my-routine
crewship routine step-override clear my-routine summarize

​

Debug a single step (step-run)

crewship routine step-run parse-invoice extract --input @sample.json
# a downstream step that consumes an upstream output — seed it with --outputs:
crewship routine step-run reconcile-invoices reconcile \
  --input @sample.json --outputs '{"parse":"{\"total\":42}"}'
crewship routine step-run parse-invoice extract --input '{"name":"a.pdf"}' --tier-override fast
crewship routine step-run fetch-and-transform normalize --input @raw.json   # transform step
crewship routine step-run notify-webhook post-it --input @payload.json     # http step

• --input is the step’s inputs fixture — a JSON object, inline or @file.json. The step’s {{ inputs.X }} refs resolve against it.
• --outputs seeds upstream {{ steps.X.output }} refs — a JSON object mapping step_id → that step’s output (a string, or any JSON value, which is stringified). Most non-first steps read an upstream output, so without --outputs those refs render empty and you’d be iterating against different input than a real run — so the command warns loudly (⚠ step references {{ steps.parse.output }} but no --outputs fixture was provided…) when a referenced upstream output isn’t seeded, across whichever fields the step type actually templates (prompt for agent_run; url/body/headers for http; input/expression for transform; args/env for script). Still no DAG traversal — the fixture stands in for upstream steps; nothing upstream executes.
• --tier-override (trivial|fast|moderate|smart) swaps the step’s tier for cheap structural iteration on an agent_run step; a step-level model_override still wins. Ignored for http/script/transform — there’s no tier to override.
• The response is marked simulated: true — it does not create a run record, so it never shows up in routine records/metrics or pollutes the run trace. A script step-run still executes for real in the crew’s container (real side effects — only the run-record bookkeeping is skipped), and an http step-run still goes through the same egress/SSRF guard a real run enforces, not a relaxed check.
• Not supported: wait (blocks on external state), notify/call_pipeline (side effects that don’t make sense without a real run), and code (needs the same sandboxed-container wiring script has, not yet extended here).

​

Agentless routines

{
  "dsl_version": "1.0",
  "name": "cost-spike-probe",
  "agentless": true,
  "egress_targets": ["billing.example.com"],
  "steps": [
    { "id": "fetch", "type": "http",
      "http": { "method": "GET", "url": "https://billing.example.com/today" } },
    { "id": "amount", "type": "transform",
      "transform": { "input": "{{ steps.fetch.output }}", "expression": ".spend_usd" } },
    { "id": "judge", "type": "code",
      "code": { "runtime": "expr", "code": "{{ steps.amount.output }} > 100" } }
  ]
}

• Save time — validation rejects agent_run (direct LLM spend), call_pipeline (the target resolves by slug at runtime, so a nested routine could gain an agent step later and silently break the guarantee), crewship (an @mention in an issue body wakes an agent, an assignment dispatches one — whether it happens depends on rendered content, not on anything provable at save time), and eval.online with sample_rate > 0 (online grading runs a grader agent against the routine’s runs).
• Run time — the executor independently refuses to dispatch an LLM-capable step inside an agentless run, so even a definition written before the validator existed fails closed.

​

Two-tier execution

{
  "trivial":  { "primary": {"adapter":"claude","model":"claude-haiku-4-5-20251001"} },
  "fast":     { "primary": {"adapter":"claude","model":"claude-haiku-4-5-20251001"},
                "fallback":[{"adapter":"claude","model":"claude-sonnet-4-6"}] },
  "moderate": { "primary": {"adapter":"claude","model":"claude-sonnet-4-6"} },
  "smart":    { "primary": {"adapter":"claude","model":"claude-opus-4-7"} }
}

​

Save validation gate

​

Governance — agent proposes, human approves the risky ones

​

Maker-checker on save

• it declares an integrations_required the author crew can’t currently satisfy (the same resolver the run gate uses);
• it has any http/egress step (or routine-level egress_targets);
• it has any code-runtime step;
• it declares credentials_required.

• Safe → active. Goes live immediately, exactly as before.
• Risky → proposed. The routine is persisted but not runnable, and a blocking inbox item is raised for MANAGER+ (the same Inbox surface as proposed skills). Approve it to go live, or reject it.

​

Approve / reject (MANAGER+)

crewship routine approve <slug>   # status → active, resolves the inbox item
crewship routine reject  <slug>   # removes the routine, resolves the inbox item

• POST /api/v1/workspaces/{ws}/pipelines/{slug}/approve — MANAGER+. Flips to active.
• POST /api/v1/workspaces/{ws}/pipelines/{slug}/reject — MANAGER+. Soft-deletes the proposed routine.

​

Disable / enable (OWNER/ADMIN airbag)

crewship routine disable <slug>   # status → disabled; cancels in-flight runs
crewship routine enable  <slug>   # status → active

• POST /api/v1/workspaces/{ws}/pipelines/{slug}/disable — OWNER/ADMIN. Flips to disabled and cancels any in-flight runs of that routine immediately.
• POST /api/v1/workspaces/{ws}/pipelines/{slug}/enable — OWNER/ADMIN. Returns it to active.

crewship routine list --status proposed

​

Draft trigger activation — a second, independent approval

​

Triggers

​

Atomic authoring — routine, version, and trigger together

crewship routine save --name daily-report --definition daily-report.json \
    --author-crew crew_marketing --cron "0 9 * * 1-5" --timezone "Europe/Prague"

Saved routine daily-report (id=..., hash=...)
Trigger: schedule 0 9 * * 1-5 — first run 2026-09-08T07:00:00+02:00
Invoke with: crewship routine run daily-report

​

Cron schedules

crewship routine schedules create --slug summarize-text \
    --name "daily-summary" --cron "0 9 * * *" --timezone "Europe/Prague" \
    --inputs '{"text":"…default"}'

​

Natural-language schedules (--when)

crewship routine schedules create --slug summarize-text \
    --name "daily-summary" --when "every weekday at 9"
# Parsed "every weekday at 9" as cron "0 9 * * 1-5" (UTC).
# Next 3 fire times:
#   - 2026-07-27 09:00 UTC
#   - 2026-07-28 09:00 UTC
#   - 2026-07-29 09:00 UTC
# Create this schedule? [y/N]:

​

Missed-run catch-up

crewship routine schedules create --slug daily-report \
    --cron "0 9 * * *" --catchup all   # backfill every missed daily run

crewship routine schedules update <schedule_id> --catchup skip

​

Wake gates

# LLM only runs when the probe prints something truthy:
crewship routine schedules create --slug cost-report \
    --cron "*/15 * * * *" \
    --wake-slug cost-spike-probe --wake-inputs '{"threshold":"100"}'

# Drop the gate later (fire on every tick again):
crewship routine schedules update <schedule_id> --no-wake

crewship routine schedules create --slug feed-change-report \
    --cron "*/15 * * * *" \
    --wake-slug feed-watch-probe

• The probe must be agentless: true, live in the same workspace, and can’t be the schedule’s own routine — all validated when the schedule is saved.
• Probe errors fail open by default: a broken or deleted probe wakes the main routine instead of going silently blind, and records last_wake_status: ERROR so you can see the probe needs fixing. This is the right default for a monitoring gate — you would rather over-fire than miss a signal.
• Fail closed for unattended gates: when the gate is meant to suppress an autonomous run — not just save tokens — a broken probe that fires the run anyway is a security hole (a tampered or unreachable probe cannot be allowed to green-light the run). Pass --fail-closed so any non-affirmative probe outcome (error, timeout, or a non-COMPLETED run) holds the run instead of firing it. The tick records last_wake_status: HELD, does not increment wake_fire_count, and advances next_run_at like a skipped tick. The default stays fail-open, so existing schedules are unchanged.
  # A probe error now HOLDS the run instead of firing it:
  crewship routine schedules create --slug nightly-cleanup \
      --cron "0 3 * * *" \
      --wake-slug maintenance-window-open --fail-closed
  
  # Flip an existing gate's policy either way:
  crewship routine schedules update <schedule_id> --fail-closed
  crewship routine schedules update <schedule_id> --fail-closed=false
  
  Note an empty or ambiguous output on a COMPLETED probe run is already treated as falsy → SKIPPED (the main run never fires) regardless of policy — --fail-closed only changes the failure branch.
• A skipped tick advances next_run_at but leaves last_run_* untouched — run telemetry stays strictly about main runs. Wake telemetry lives in wake_check_count / wake_fire_count / last_wake_at / last_wake_status, and routine schedules list shows it as <probe> woke/checked in the WAKE column.
• Probe executions are regular runs with triggered_via: wake_check, so they’re auditable in run history and filterable out of dashboards.

​

Circuit breaker

# Trip after 3 straight failures instead of the default 5:
crewship routine schedules create --slug nightly-report \
    --cron "0 6 * * *" --max-failures 3

# Change the threshold on an existing schedule:
crewship routine schedules update <schedule_id> --max-failures 10

• The schedule is disabled (enabled: false) with disabled_reason: circuit_breaker — distinct from an operator-initiated disable (disabled_reason stays empty), so routine schedules list and routine doctor can tell you why a schedule went dark.
• A journal event (pipeline.schedule.circuit_breaker_tripped) and exactly one actionable MANAGER inbox alert fire for the trip itself — separate from the per-run failed_run alerts, which still fire once per failed run.
• A successful fire resets the consecutive-failure counter to 0, so an intermittently-flaky routine that eventually succeeds never trips.

crewship routine schedules enable <schedule_id>

​

The reliability editor

• When — cron expression + timezone, with a live preview of the next five fire times computed server-side as you type (see below).
• After downtime — the catch-up policy.
• Only run if — the wake gate, including fail-open/closed.
• On repeated failure — the circuit breaker threshold.
• Version — the pin, or track head.
• Health — last run, next run, consecutive failures, disabled reason, missed count and wake stats stay read-only on the list row; they are telemetry, not settings.

​

A plan’s preset is checked when the plan is written

​

Next-fire-times preview

crewship routine schedules preview --cron "30 2 * * *" --timezone Europe/Prague --count 5

​

Missed occurrences on recovery

​

Webhooks

crewship routine webhooks create --slug pr-review-structured \
    --hmac-secret "$(openssl rand -hex 32)" --rate-limit 30

ts=$(date +%s)
sig=$(printf '%s.%s' "$ts" "$body" | openssl dgst -sha256 -hmac "$secret" | cut -d' ' -f2)
curl -X POST "https://.../api/v1/webhooks/$token" \
  -H "X-Crewship-Timestamp: $ts" \
  -H "X-Crewship-Signature: sha256=$sig" \
  -d "$body"

• Any agent_run step’s rendered prompt wraps those three input values in the same untrusted-ingress fence (<untrusted source="webhook" ...>) the agent-webhook path already uses — the model is told to treat fenced content as data, never instructions. Other template destinations (http URL/body/headers, code, transform, if) are not fenced — wrapping those would corrupt a URL or a JSON body rather than protect anything; they’re covered by the egress hardening below instead.
• Every http step in a webhook-triggered run is held to the crew network policy’s restricted floor regardless of the crew’s actual network_mode — see Egress enforcement above.

HTTP/1.1 202 Accepted
{"run_id": "run_c...", "status": "PENDING", "deduped": false}

crewship routine result <run_id>        # just the deliverable (the run's final output)
crewship routine logs <run_id>          # persisted run state (status, step outputs, error, output)
crewship routine logs <run_id> --full   # full step-by-step journal timeline
# or: GET /api/v1/workspaces/{wsId}/pipeline-runs/{runId}

​

Client-facing views

• Live progress — crewship routine watch <slug> --progress collapses the raw event tail into one updating status line: ▶ reconcile-invoices · step 2/4 (verify) · RUNNING · $0.0021 · 12s. Step count comes from the routine definition; cost + elapsed from the run’s events. It exits when the run finishes. (Without --progress, watch stays the ANSI event tail for CI/scripting.)
• Shareable report — crewship routine report <run_id> assembles a run into a readable document: inputs → each step’s outcome + output → the final deliverable → cost & duration. Markdown by default (paste into a ticket/chat); -f html -o run.html writes a self-contained page to hand to a customer (output is HTML-escaped — no injection from run data).
• Redacted mode — --client on result, report, and logs drops the internal noise (run-id, cost, tier, mode, trigger, entry_type labels) and renders status in plain words (Succeeded / Failed), leaving just the routine name, inputs, and deliverable.

crewship routine watch reconcile-invoices --progress
crewship routine report run_abc123 -f html -o run.html
crewship routine result run_abc123 --client        # plain: "Acct — Succeeded" + the output

​

Editing a webhook in place (F21)

crewship routine webhooks update wh_abc123 \
    --name "github-pr-reviews-v2" --rate-limit 120 \
    --inputs-template '{"source":"github"}'

crewship routine webhooks update wh_abc123 --rotate-secret

​

Manual

​

Deferred dispatch: delay, ttl, debounce, priority

# fire 60s from now, expire if not dispatched within 5 min, high priority
crewship routine run my-routine --delay 60 --ttl 300 --priority 9

# coalesce a burst: repeated triggers sharing a key collapse into one run
crewship routine run my-routine --debounce-key vendor-42 --debounce-window 30 --debounce-max 300

crewship routine pending list            # not-yet-fired deferred triggers
crewship routine pending cancel <id>     # cancel before it fires

​

Dry-run preview

crewship routine dry-run email-fetch --inputs '{"since":"yesterday"}'

​

Versioning + rollback

crewship routine versions email-fetch
crewship routine rollback email-fetch --to 3

​

Diffing versions

crewship routine diff email-fetch --from 2 --to 3

​

The improvement loop: routine iterate

crewship routine iterate summarize-text \
  --rubric ./rubric.md --inputs '{"text": "..."}' \
  --grader reviewer --optimizer lead --author-crew eng

• Budgets are enforced, not advised — every run and agent call lands in the Paymaster ledger under workspace budgets.
• The gates don’t move — each save passes the same server-side test-run gate and governance rules as a human author; a definition that gains risky steps (http, code, egress) parks as proposed for review and the loop stops.
• Nothing is lost — full journal per run, one version row per accepted round, reversible by design.

​

Bundle export / import

# Export from workspace A
crewship routine export email-fetch --include-history > email-fetch.json

# Import into workspace B — --crew names the author crew that will own it
cat email-fetch.json | crewship routine import --crew acct

​

HITL waitpoints

# CLI
crewship routine waitpoints list
crewship routine waitpoints approve <token> --comment "LGTM"
crewship routine waitpoints reject  <token> --comment "needs revision"

# UI: the run's Activity panel (History → run) or /inbox → Approve / Reject
# API: POST /api/v1/workspaces/{ws}/pipelines/waitpoints/{token}/approve

​

Durability and restart recovery

• Completed steps are restored, not re-executed. Their outputs feed downstream {{ steps.X.output }} templates exactly as if the process had never died.
• The in-flight step re-executes from scratch — at-least-once semantics. For an agent_run step this means the agent call is re-issued (and re-billed); http/code/script steps with external side effects may fire twice. Design steps to be idempotent where that matters.
• Runs parked on a wait approval step re-attach to the original waitpoint token. No duplicate approval card is created; the pending inbox item stays answerable across the restart, and approving it resumes the run.
• Runs parked on a wait: event step re-attach the same way (#1409): the wait durably ARMs a pipeline_signal_waits row before parking, so a signal delivered via POST .../signal at any point — including while the process was down — lands in that row rather than an in-memory-only channel. Resume checks for an already-delivered payload first; pending top-level waits park again with the original deadline, and elapsed deadlines produce the normal timeout failure. A valid consumed payload can be replayed if a crash happened before the step output was saved.
• DAG runs (needs:) resume at wave granularity — the parallel scheduler flushes state when each wave completes, so a kill mid-wave re-executes that wave’s unfinished steps.
• call_pipeline boundaries are NOT persisted. A kill while a nested pipeline is executing re-runs the entire nested pipeline on resume — the parent’s call_pipeline step is the unit of recovery, and the nested run’s own per-step progress is not checkpointed. Keep nested pipelines short or idempotent if a mid-flight kill matters to you.
• The accumulated cost is restored too, so max_cost_usd keeps counting across the restart instead of resetting. Caveat: cost is flushed at step boundaries, so whatever the killed in-flight step had already spent before the kill is not in the restored total — the cap under-counts the true spend by up to one step’s worth (and the re-executed step is billed again on top).
• A resumed run that finds its concurrency slot occupied waits and retries with capped exponential backoff (2s doubling up to 60s) instead of failing — losing the slot race to a freshly-fired scheduled run is a timing collision, not a reason to abandon hours of restored work. If the server shuts down while a run is still waiting for its slot, the row stays in-flight and the next boot resumes it again.
• A waitpoint that timed out while the process was down resumes, observes the expired token, and fails with wait step "X" (approval) timed out — distinct from an operator clicking deny (… denied).

​

Online eval sampler

name: nightly-summary
eval:
  online:
    sample_rate: 0.05              # grade 5% of runs; 0 disables, 1.0 grades every run
    grader_agent_slug: qa-grader   # references an agent in the AUTHOR crew with a rubric prompt

1. Scan pipeline_runs WHERE status = 'completed' AND completed_at > watermark.
2. For each candidate, resolve the routine DSL. If eval.online is absent or sample_rate <= 0, skip.
3. Draw from crypto/rand. If the sample lands above sample_rate, skip.
4. Otherwise, INSERT into eval_runs with kind = 'online', status = 'queued'. The existing grader worker picks it up and writes the result back.

• Schema-layer idempotency — eval_runs has a partial UNIQUE INDEX (pipeline_run_id) WHERE kind='online'. A duplicate sampler instance or a crash-recovery watermark replay can attempt the same enqueue twice; the second collapses to a no-op rather than queueing twice and double-billing the grader.
• (completed_at, id) tuple cursor — parallel fan-out steps that complete at the same nanosecond all get graded; a timestamp-only cursor would orphan siblings.
• Stuck-on-error watermark — a transient per-row error (resolver outage, entropy outage, enqueue conflict) freezes the watermark at the row before the failure so the next tick retries. Deterministic skips (no eval config, sample roll missed) advance normally.
• Trace correlation — the eval row carries routine_slug + pipeline_run_id so an operator clicking a low-scoring grade in the eval UI lands on the actual trace via pipeline_run_id -> trace.

• Watermark is in-memory only. On process restart it resets to now - 1h; outages longer than that leave a gap in grading coverage. The UNIQUE index keeps reprocessing harmless.
• Page cap is 500 rows/tick. A workspace processing > 500 routine runs/minute at sample_rate = 1.0 will accumulate backlog; the watermark only advances past successfully-handled rows, so the backlog drains across subsequent ticks rather than being lost.

​

Validation gates and credential leak guards

• must_not_contain: ["API_KEY=", "Bearer "] — output must include none of these substrings
• must_contain: ["##"] — output must include all of these
• min_length / max_length — convenience for non-JSON outputs

​

Observability

• pipeline.run.started — once at run begin
• pipeline.step.started — per step
• pipeline.step.container_ready — per agent step: how long the step spent acquiring its crew container (duration_ms). This isolates the container-provision cost from the LLM/tool time in the step’s total duration — so you can read the effect of container prewarming directly (a warm hit is near-zero, a cold provision is seconds) instead of guessing under LLM-latency noise. A transiently-retried step emits one record per attempt, distinguished by an attempt field in the payload. Visible in routine logs <run_id> --full (kept off the compact activity rail).
• pipeline.step.completed / pipeline.step.failed / pipeline.step.validation_failed — per step terminal
• pipeline.step.skipped — the step’s if condition was false, so it never ran (payload carries the condition). A dedicated entry type rather than a completed row with a marker, so a skipped branch is distinguishable from a real completion at the storage layer.
• pipeline.step.retrying — a transient failure the retry policy is about to swallow; non-terminal, one per attempt (attempt/max), followed by the step’s eventual completed/failed. (Rows written before this type existed arrive as pipeline.step.completed/failed carrying a kind=skipped/kind=retry payload marker; readers honour that marker as a fallback.)
• pipeline.run.completed / pipeline.run.failed — once at run end
• summary.generated — an LLM-generated outcome verdict (“did the run accomplish the goal?”), fired once after the run terminates. Shown as an outcome pill next to the status pill in the Runs sub-tab. Gated by the run_verdict_summaries feature flag and skipped for agentless/dry-run routines — see the Crew Journal guide for the full contract.

​

Status, outcome and verdict

​

The step’s chat is a durable transcript

​

Live visibility — what is a routine doing right now?

• Header chip — a pulsing “N routines running” pill appears in the toolbar next to the Online / Crews pills (hidden when nothing is active). If any run is parked on a human approval it turns amber and appends ”· M awaiting approval”. Clicking opens a popover with the six newest active runs — routine name, short run id, elapsed time, cost so far, current step — each with Open trace ↗ (deep-link to /activity?run=<id>), Cancel (same manage-tier RBAC as the Runs tab), and a Review → shortcut into the routine for parked runs. With more than six active runs a “View all N running →” footer jumps to the Activity rail pre-filtered to the active bucket (/activity?status=active).
• /routines list — a routine with an active run shows a live Running pill, and a parked run Waiting for you with its own banner above the list.
• /routines list table — the status cell swaps the historical “last run” pill for a live Running (or amber Awaiting approval) pill with current step · elapsed · cost, and live routines bubble to the top of the table regardless of the chosen column sort.

​

Run warnings

{ "stage": "hook after_all", "message": "credential release step timed out", "at": "2026-07-02T12:01:05Z" }

​

Per-step cost + duration

• UI: right-aligned columns next to each step row, with a footer total summing the run. An em-dash (—) renders for events that don’t carry cost (pipeline.step.started, .failed, live-only echoes) — easier to scan than $0.0000 next to real values.
• CLI logs: extra DURATION and COST columns in the timeline output. Same em-dash rule for non-positive values.

​

Monthly budget meter

crewship routine budget set my-routine --amount 50    # cap at $50/month
crewship routine budget get my-routine
# my-routine (2026-07): [#####---------------]
#   $12.50 of $50.00 (25%)
crewship routine budget set my-routine --amount 0      # clear the cap
crewship routine budget summary                        # workspace roll-up

$ crewship routine budget summary
ROUTINE          BUDGET   SPENT   PCT   STATUS
my-routine       $50.00   $12.50  25%
cost-report      $20.00   $22.10  111%  OVER
summarize-text   —        $3.40   —

2026-07 total: $37.99 spent of $70.00 budgeted

​

Run tags & metadata

crewship routine run cost-spike-probe \
  --tag prod --tag billing \
  --metadata '{"source":"manual","ticket":"OPS-42"}'

​

Replay & error fingerprinting

crewship routine replay <run_id>                   # re-invoke with the run's captured inputs
crewship routine errors                             # list fingerprint groups
crewship routine bulk-replay --fingerprint <fp>      # replay every run in a fingerprint group after a fix

​

RBAC

​

Common patterns

​

”Run this routine every day at 9 AM"

crewship routine schedules create --slug daily-status-digest \
    --cron "0 9 * * *" --timezone "Europe/Prague"

​

"Trigger this routine from a GitHub Actions hook"

crewship routine webhooks create --slug pr-review-structured \
    --hmac-secret "$(openssl rand -hex 32)" --rate-limit 30
# Use the printed token + secret as Settings → Webhooks in the GH repo.

​

"Validate routine DSL in CI before committing”

- run: crewship routine validate routines/*.json

​

”Discover what an agent is about to do, without running it”

crewship routine dry-run summarize-text --inputs '{"text":"..."}'

​

”Watch a metric on a tight cron, spend tokens only on a spike”

crewship routine schedules create --slug incident-summary \
    --cron "*/5 * * * *" --wake-slug error-rate-probe

​

“Get a daily ops digest of runs, cost, and failures”

crewship digest enable --crew ops

​

”Cancel an in-flight run”

crewship routine cancel <run_id>

​

”Serialise runs per tenant / customer / repo”

{
  "concurrency_key": "{{ inputs.account_id }}",
  "max_concurrent": 1,
  "inputs": [
    { "name": "account_id", "type": "string", "required": true }
  ]
}

​

Troubleshooting

pipeline: concurrency_key rendered to empty value (referenced input missing or empty): template "{{ inputs.<NAME> }}"

​

CLI reference

​

Backend reference

• Migrations v78 (pipelines + execution_tiers_json), v79 (versions + waitpoints), v80 (schedules), v81 (run idempotency), v82 (webhooks), v115 (schedule wake gates), v153 (schedule circuit breaker), v154 (signal waits)
• Source: internal/pipeline/ (~10 700 LOC, 36 files)
• API: internal/api/pipelines.go, pipeline_runs.go, pipeline_schedules.go, pipeline_webhooks.go
• Sidecar: internal/sidecar/pipelines.go (port 9119)
• Frontend: app/(dashboard)/routines/, components/features/routines/, hooks use-pipelines*

​

Production notes