Skip to main content

​
What is a Routine?

A Routine is a declarative recipe for repeatable AI work. You author it once (or an agent authors it for you when it spots a repetitive pattern), it runs the same way every time, and it lives in your workspace as a reusable asset alongside crews, skills, and credentials. Each routine bundles:
  • A JSON DSL definition — inputs, outputs, ordered or DAG-structured steps, validation gates, declared egress, credential requirements
  • Authorship metadata — which crew, which agent (or user), via which path
  • Triggers — cron schedules and webhook tokens that fire it autonomously
  • Run history — every invocation as an immutable journal trace with step-level events
  • Versions — every save creates a new version; rollback is one API call
Compared to other layers in Crewship:

​
The routines page

Two states, no tabs: the overview, or the routine you picked in the sidebar. The overview answers, in the order you ask on arrival — did anything run today, is the catalog healthy, what fires next, what ran, what did it cost, what is broken: There is no Refresh button, because a dashboard with one is a dashboard admitting it is not live. Run events already pushed themselves; the catalog now broadcasts pipeline.saved on every save, approval, rejection, disable and delete, and the schedule list listens to run events too — next_run_at moves the moment a schedule fires. On a phone the routine sidebar opens over the overview rather than beside it: 280px of a 390px screen is not a column, it is a replacement. Picking a routine closes it. | Runs · 7 days | The week by verdict — green passed, red failed, amber not yet judged, grey cancelled. The bars sum to the runs that day: a cancelled run is stacked rather than dropped, because a bar shorter than its own day is a chart lying by omission. | | Recently failing | The last failed runs, each naming the step it died on and the error. One routine failing three times is three rows — that repetition is the signal. Each opens the run in Activity. | No money on this page. Crewship bills by subscription and is driven from the CLI, so a per-run dollar figure is an internal estimate dressed as an invoice — four decimal places about a number nobody is charged. The author’s estimate (estimated_cost_usd) stays on the routine, where “roughly what will this cost me to run” is a real question; summing it into a workspace ledger answered one nobody asked. This replaced a List / Schedules / Insights tab bar. List rendered a table of every routine beside a sidebar that was already the catalog — the same list twice, and the copy in the main pane was the one you could not search. Schedules was a read-only table; every action on a schedule (create, pause, delete) lives on the routine’s own Triggers card, so the tab held no capability. Insights was four derived numbers and a “top routines by usage” leaderboard. What was load-bearing in those two — what fires next, what runs cost, which routines are over budget, what is failing — is a card on the overview now. New routine offers three ways to fill one buffer, because the server has one way to create a routine: test_run mints a save_token bound to the definition’s hash, and save refuses without a fresh one. Describe it hands off to a crew Lead carrying the routine-author skill. Fork copies one of your own routines’ definitions, renamed. Write it yourself opens the same editor the detail page uses — YAML by default, schema completion, errors located on the line, and the step graph beside the code. All three end at the same Test-run → Save. The skip test-run gate checkbox appears only for OWNER/ADMIN, which is the tier the server accepts it from. An affordance that cannot work reads as a broken product rather than a limit on the person.

​
The detail page

Opening a routine gives you one scrolling surface of cards, not a tab bar: A routine’s monthly budget is on its own card here. It used to be a workspace-wide roll-up on the overview, which put a third card about money on one row — and before that a tab nobody opened. It is kept as a per-routine guardrail; it is not a bill, and nothing on the overview sums it. A routine also carries an icon and colour, picked from the header and stored in columns of their own — so recolouring never touches the definition, never mints a version, and never invalidates a save token. Unset routines derive a stable icon from their slug, so a list of thirty is legible before anyone chooses anything. CLI: crewship routine appearance. Wait points are not on the detail page: they belong to a run, and Activity is where the run is. The pending ones are on the overview’s Waiting on you card, decidable in place — a parked run is a stopped process, and finding it should not require knowing which routine stopped. A routine awaiting approval says what it is asking for, not just that it asks: the banner lists the credentials, integrations and egress hosts its definition declares, beside Approve, Reject and an Inbox button that deep-links the review row itself. credentials_required is the risk category; github:repo is the ask, and it is the one a reviewer actually weighs. The same list is on the inbox item, and so are Approve / Reject. They were missing: a routine proposal fell through to “no decision to make here” while the routine sat at proposed, unable to run, with both endpoints live on the server. The queue said “your move” and the item said “not here”. When a routine does land as proposed, its inbox review item carries a unified diff of the two versions — what was last accepted against what is being proposed — so a reviewer decides from the change itself rather than from a slug and a risk label. A routine’s first version has no predecessor, so that item says so instead of diffing against nothing. One item per routine, refreshed each time that routine goes for review: proposing again reopens it with the current versions, reasons and diff rather than adding a second card. That matters most on the second pass — a routine reviewed once, approved, then edited into new risk raises the review again instead of being swallowed by the resolved item from last time. There is no Dry run button. Two thirds of what its report showed — the step plan and the declared resources — is the Definition graph and the Access card, permanently on the page. Its one unique fact, the model each agent step resolves to, is now a chip on the node it describes, and the author’s cost estimate is in the figures strip. The /dry_run endpoint and crewship routine dry-run are unchanged and still the right thing in CI, where nobody is looking at a page. The user-facing label is Routine across web UI, CLI, agent system prompts and marketing surfaces. The backend identifier (database table, Go package, HTTP route paths) remains pipelines for backwards compatibility.
Naval theme. Routines are the boring, accurate, repeated procedures a ship’s crew runs every day — the equivalent of naval drills in the Crewship metaphor. Programs your AI agents follow.

​
Three authoring paths

Crewship is unusual among workflow systems because agents author routines, not just execute them. Three paths converge on the same pipelines table:

Agent

The most common path. An agent that’s solved a repetitive problem twice posts to http://localhost:9119/pipelines/save from inside its container; the sidecar injects authorship and forwards to the main API. Next time [AVAILABLE ROUTINES] block in the system prompt advertises it to other crews.You can also just describe a routine in chat — “make a routine that summarizes yesterday’s commits and posts to Slack.” A crew Lead carries the bundled routine-author skill (an authoring playbook): it clarifies the essentials, grounds the DSL in the crew’s [CONNECTED INTEGRATIONS] and [AVAILABLE ROUTINES], writes and test-runs the routine, then tells you whether it went live or landed as a proposed routine for a Manager to approve (see Governance).

UI

Open /routines, click + New routine, pick a starter template, edit the DSL JSON, click Test & Save. Test_run runs against the execution tier; on pass the routine is persisted with authored_via=user_api and the JWT user as author. An existing routine’s DSL is also fully editable in-place from its detail page — press Edit code on the Definition card and the editor opens beside the graph (CodeMirror, format + revert + copy) — see the note below on how that path’s Save differs from Test & Save.

CLI

Scaffold a starting file with crewship routine init -o file.json (a minimal valid one-step skeleton) or clone an existing routine with crewship routine init --from <slug> -o file.json. Then crewship routine save --name "..." --definition file.json --author-crew <crew-slug-or-id>. The server validates the DSL on save (same gate as the UI). CI-friendly: validate offline first with crewship routine validate file.json, then save. Print the machine-readable authoring contract with crewship routine schema (wire it into an editor for autocomplete, or hand it to an agent authoring a routine).
Editing an existing routine’s DSL bypasses the test-run gate. The Definition card’s editor Save button posts straight to /pipelines/save with skip_test_gate: true — it does not send the definition through test_run first. The server only honors that flag for OWNER/ADMIN (lower roles get 403), so this is a fast lane for trusted operators, not a hole in the gate: a MEMBER/MANAGER can view and copy the DSL but can’t save an edit without going through the create flow’s Test & Save. A follow-up will chain test_run → save_token → save behind one button so any MANAGER+ role can edit safely.

​
Authoring with AI (one-shot)

Handing a routine to an LLM to author (Claude Code over the CLI, or an in-container agent) used to mean piecing the palette together from ~8 commands — routine schema for the DSL, crew config for the container, separate calls for integrations, agents, and runtimes — and still guessing which runtimes are actually wired. crewship routine capabilities <crew> collapses that into one discovery dump:
The bundle carries everything needed to write a routine validate-clean DSL on the first try:
  • schema — the routine DSL JSON schema (the same bytes as routine schema), nested inline so the whole contract travels in one response.
  • container — the crew’s resolved devcontainer capabilities: datastores (host/port to declare under resources.datastores) and installed CLI tools.
  • integrations — connected integrations WITH their enabled tool names (GMAIL_FETCH_EMAIL, …), so the author references real tools, not guesses.
  • agents — the crew’s agent slugs, which agent_run steps reference.
  • runtimes — the truth about what’s wired in this build: code.wired (expr, cel) vs code.reserved_unwired (python/go/bash — legal names but no runner, so don’t use them), and the type: script interpreter inference table (.py → python3, …).
In-container parity. An agent working inside a crew container gets the identical bundle via the discover_capabilities MCP tool on the crewship-routines server (alongside save_routine / list_routines / run_routine) — scoped to its own crew, no arguments. Call it first when authoring so save_routine passes its test-run gate on the first attempt. The in-session [CONNECTED INTEGRATIONS] system-prompt block also now names each integration’s enabled tools, so an agent answering a live request knows exactly which tools to reach for. Backed by GET /api/v1/crews/{crewId}/capabilities.

​
DSL anatomy

Minimal valid routine:

​
YAML instead of JSON

crewship routine validate and crewship routine save --definition also accept YAML — sniffed from the file’s content, not its extension, and converted to canonical JSON before validation/upload. The same minimal routine, as YAML:
The main win is a real multiline prompt via YAML’s literal block scalar (|) instead of a JSON string full of \n escapes — this is also why routine init’s scaffold used to smuggle its “run routine schema, replace agent_slug” pointer into the description field (JSON has no comment syntax); with YAML input that workaround is no longer necessary, though the JSON scaffold itself is unchanged. The API, the DB, and every other Parse call site still only ever see canonical JSON — YAML is purely a convenience at these two CLI entry points, not a second stored format.

​
Top-level fields

​
InputSpec

type is one of string | integer | number | boolean | array | object. min / max are *float64 so decimal constraints work for number types.

​
Template substitution

Anywhere a string is interpolated (prompt, http URL/body/headers, wait until, code, transform, conditional if), placeholder {{ ... }} resolves against:
  • inputs.X — declared input value
  • steps.Y.output — full text output of an earlier step
  • steps.Y.output.path — JSON path into a step’s output (when output parses as JSON)
  • env.AUTHOR_CREW_NAME / etc. — read-only allowlist of execution context
  • secrets.<type> — a workspace-vault credential, resolved at run time by type (see Secrets below)
  • routine.state.<key> — durable cross-run state carried forward from a previous run (see Cross-run state below)
Save-time validator walks every template-bearing field (prompt, nested inputs, http url/body/headers, wait until, event_filter, approval_prompt, code body, code env values, transform input, transform expression, if condition) and rejects placeholders that reference unknown inputs or unseen-yet steps.

​
Secrets ({{ secrets.<type> }})

A code / script / notify / http step that needs an API key, token, or password references it as {{ secrets.<type> }} — never by pasting the value into an env: map (which is versioned, exported, and diffed in plaintext). At run time the placeholder resolves to the decrypted value of a matching ACTIVE credential in the workspace vault, resolved by type exactly like an http step’s credential_ref: workspace-scoped, author-crew isolated, newest-active-wins on rotation.
The value never lands in the versioned DSL (the definition carries only the {{ secrets.stripe }} template), and it is scrubbed from the step’s downstream output, the run journal, script-audit command lines, notification bodies, and error messages — so an echoed or reflected secret shows up as [REDACTED:secret], not the real value. A type with no ACTIVE credential renders empty (like a missing input), so a public/optional call keeps working; to make an unresolvable credential a hard failure, declare it in credentials_required.
Because secrets are scrubbed from step outputs, you cannot thread a secret between steps as {{ steps.X.output }} — each step re-resolves {{ secrets.<type> }} fresh. This is deliberate: a plaintext secret must never flow through the step-outputs record.
Templates are regex substitution, not expression evaluation. There is no arithmetic, no function calls, no inline conditionals. If you need logic, add another step.
Save-time validation walks every template-bearing field and rejects placeholders that reference unknown inputs or unseen-yet steps — so a typo’d {{ inputs.tyop }} fails at save, not at 3am in production.

​
Cross-run state ({{ routine.state.<key> }})

run.metadata.* is per-run scratch — gone when the run ends. Cross-run state carries a value forward to the next run of the same routine, which is what every incremental / polling job needs: a watermark (“last processed id / timestamp”) so each run picks up where the last left off.
  • Read with {{ routine.state.<key> }}. A key that was never written renders empty (like a missing input). The value you read is the snapshot as of run start — it reflects what a prior run wrote, not a write earlier in this same run.
  • Write with a step’s state_write binding — a map of key → template, rendered after the step completes (so it can reference the step’s own output) and persisted for the next run:
State is isolated per schedule: two cron schedules of the same routine keep independent watermarks (keyed on (routine, schedule)); manual / webhook runs share a single default bucket per routine. It is durable — a value written before a restart is read after it — because it lives in the database, not memory. A wake gate probe reads the same bucket its main scheduled routine writes, so an agentless probe can gate on the last watermark.
State is a small key/value store for coordination values (cursors, timestamps, counts), not a data warehouse. Writes are best-effort — a failed state write logs a warning and never fails the step. Values are opaque strings; JSON-encode a structure yourself if you need one.

​
Inspecting and repairing state

A watermark can go wrong in a way that is silent and permanent: write a cursor that is too far ahead (a future timestamp, an id from the wrong source) and every later run finds nothing to do — and reports success. crewship routine state is the operator surface for that.
state list shows every bucket by default — because state is isolated per schedule, the stuck cursor is usually in a bucket you would not have guessed. The UPDATED column is normally the tell: a cursor frozen three days ago next to schedules that ran this morning. Mutations are ADMIN-tier. A watermark governs what every future unattended run does, so rewriting one carries the same blast radius as disabling the routine. clear is deliberately bucket-scoped — there is no “wipe every schedule” form, because that makes each schedule reprocess its whole backlog with no undo. Prefer set to a known-good value when you can. Full flags: crewship routine state.

​
Required integrations

A routine can DECLARE the third-party integrations (Composio connectors) it needs with the top-level integrations_required array, and the run path will block a run when the executing crew hasn’t connected one of them. This closes the “integration forgotten” gap — the same shape as egress_targets, but for connectors instead of hosts.
Semantics:
  • Declared is always allowed. Saving a routine that names an integration the crew lacks is fine — declaring is a contract, not a connection. Only the run enforces. (Save-time validation only checks the list is well-formed: non-empty slugs, lowercased/trimmed, within a sane count cap.)
  • Enforced at run time. Before a run starts, Crewship resolves the integrations the routine’s author crew has connected and compares them to integrations_required. If any are missing the run is blocked with an RFC 7807 Problem Details response, HTTP 422, carrying a machine-readable missing_integrations: string[] member and a human detail like routine requires integration "slack" not connected for crew "Marketing". The UI uses missing_integrations to render a Connect action. The run never starts — no tokens spent.
  • No-op fast path. An empty / absent integrations_required does zero resolution work — no overhead for routines that don’t use it.
  • Fail-open. If integration-availability resolution itself errors, the run is allowed (a warning is logged). A bug in resolution must never wedge every run of every routine — a forgotten integration is a soft, recoverable failure; a hard block on all runs would be a self-inflicted outage.
  • run is gated; dry_run is not. A live run executes against the author crew’s agents, so it’s gated (fail fast rather than land an unrunnable routine). The internal save gate’s draft validation applies the same integration check. dry_run is a preview that invokes nothing, so it’s left ungated — it shows what the routine would need, even integrations the crew hasn’t connected yet.
Integration availability is resolved from the crew’s connected Composio connectors (the MCP server rows the bind flow writes). Two limitations: under the workspace default connector (every agent inherits all connected apps), the gate treats integrations as available without enumerating them; and resolution reflects what’s wired, not live connection health (a revoked account still reads as available until its binding is removed).

​
Required credentials

A routine DECLARES the vault credentials its {{ secrets.<type> }} refs (and http credential_refs) need with the top-level credentials_required array of { "type": "...", "scope": "..." }. Like integrations_required, the type is matched against the workspace vault by purpose, not by ID, so a marketplace “stripe” template runs against any workspace holding a stripe credential.
Semantics (mirroring Required integrations):
  • Declared is always allowed. Saving/importing a routine that names a credential the vault lacks is fine — you can connect it later. Save-time validation only checks the list is well-formed (non-empty type, deduped, within a count cap).
  • Enforced at run time. Before a live run starts, Crewship probes the author crew’s workspace vault for an ACTIVE credential of each declared type. If any are missing the run is blocked with an RFC 7807 Problem Details response, HTTP 422, carrying a machine-readable missing_credentials: string[] member and a human detail. No tokens spent, no half-run with an opaque auth failure deep in a step.
  • No-op fast path. An empty / absent credentials_required does zero probe work.
  • Fail-open on infra, fail-closed on absence. A probe error (DB hiccup) allows the run with a warning; a confirmed-missing credential blocks it.
  • Enforced wherever integrations_required is. Live run, run_batch, internal runs, and the test_run save-preview gate all check it — the same precondition boundary that already probes for required integrations and resources. Only the executor’s pure dry_run mode — which invokes no steps and resolves no secrets — is exempt. Persisting a well-formed draft is still fine (see above); it’s running and the test-run preview that block until the credential is connected.

​
Capability manifest

Every routine has a derived capability manifest — the full “what this routine touches” blast radius. The detail API (GET .../routines/{id}) returns it under a manifest member so the UI can render a data-flow diagram and governance can reason about the whole footprint of a run, not just its visible steps. Most of the manifest is auto-derived from the DSL — you don’t declare it: The walk covers routine-level (before_all / after_all / on_failure) and per-step (before / after) lifecycle hooks too — a capability hidden in an on_failure hook is still part of the blast radius. Every list is deduped + sorted and always rendered as [] (never null), so the diagram is stable.

​
The resources block

Two things can’t be inferred from the step graph: datastores a routine reads/writes, and CLI tools/scripts it runs. In production, code steps aren’t wired — agents run scripts (ansible, kubectl, …) via an agent_run step that shells out, so static analysis can’t see them. Declare them so they show up in the manifest:
  • datastores[].type — any string, but use the canonical vocabulary redis | postgres | mysql | mongodb | other so it matches what the crew’s container catalog reports (the precondition gate compares type case-insensitively; a non-canonical type simply won’t match a crew resource).
  • tools[].type — any string; canonical examples ansible | terraform | kubectl | bash | python | other. Same matching rule as datastores.
  • name / note are free-form and optional.
Validation is lenient: declaring a resource is always allowed at save time (declaring is a contract, not a provisioning request). Save only rejects malformed entries — a blank or non-slug type, or more than 32 of either kind. The run path enforces availability — see Required resources.

​
Required resources

The resources block is a run-time precondition gate, the resource sibling of Required integrations: it states what the routine requires (datastores like Postgres/Redis, CLI tools like ansible), and the run path checks it against what the executing crew’s container actually has ([CONTAINER RESOURCES] — the crew’s sidecar datastores + installed tools). Semantics:
  • Enforced at run time. Before a run starts, Crewship resolves the author crew’s container resources and compares them to the declared resources.datastores + resources.tools. If the routine requires a datastore or tool the crew doesn’t have, the run is blocked with an RFC 7807 Problem Details response, HTTP 422, carrying a machine-readable missing_resources: [{ kind, type, name }] member (kind is "datastore" or "tool") and a human detail like routine needs datastore postgres, tool ansible, not available to crew "Ops". The run never starts — no tokens spent — until the resource is provisioned on the crew.
  • Matching. A required datastore is satisfied when the crew has a datastore of the same engine type (case-insensitive; the service name is advisory and not matched). A required tool is satisfied when the crew has an installed tool whose name matches the required type (the tool’s name, e.g. deploy.yml, is the concrete artifact and isn’t matched against the container).
  • No-op fast path. A routine that declares no resources block does zero resolution work. Only the declared resources are gated — code-step runtimes folded into the manifest’s tools (e.g. cel) are internal executor runtimes, not crew CLIs, and are never gated.
  • Fail-open. If the routine has no author crew, or resource resolution itself errors, the run is allowed (a warning is logged) — same reasoning as the integration gate: a resolver bug must never wedge every run.
  • run is gated; dry_run is not. (The internal save gate’s draft validation applies the same resource check.)

​
Agents already know what the container has

You don’t have to tell an agent that its crew runs Postgres or ships kubectl — Crewship surfaces it automatically. Every agent’s system prompt now includes a [CONTAINER RESOURCES] block listing the crew’s datastores (derived from the crew’s sidecar services — a service named postgres is reachable at host postgres on its declared port) and installed CLI tools (derived from the crew’s devcontainer features + mise toolchain, e.g. ansible, kubectl, git, python, node). The agent is instructed to use these directly instead of probing or trying to install them. So when you author a routine that needs a datastore, declare it under resources.datastores and connect via the host/port the agent already sees in that block. The block is omitted entirely when a crew has no services and no notable tools.

​
Step types

​
agent_run

complexity resolves through the workspace’s execution_tiers_json mapping into (adapter, model). With complexity: "fast", the agent’s CLI gets --model claude-haiku-4-5-20251001 (or whatever the workspace mapped fast to). model_override is the explicit pin that wins over complexity. on_fail lives at the step level (not inside validation) and is one of escalate_tier | abort | retry_step — escalate_tier walks the fallback chain (e.g., Haiku → Sonnet → Opus) until validation passes or the chain exhausts. retry_step is sugar for the default retry policy (retry the step on a transient execution error); for the validation/rubric gate it behaves as escalate_tier.

​
Retry transient failures

retry is a per-step policy for execution errors — the runner failed before an output could even be validated (HTTP 5xx, a code-step timeout, a rate limit, a network blip). It is distinct from on_fail, which handles a bad output that failed its validation/rubric gate. Retries exhaust first; then on_fail engages.
retry_on sees three variables: error (the message string), transient (a bool from the built-in 429/5xx/timeout/net-blip classifier), and status (the HTTP status an http step failed with — parsed from its HTTP <code> error, 0 for any other error, so a stray number elsewhere in a message is never misread as a status). It must evaluate to bool. Examples: 'transient' (shorthand), 'status == 429', 'status >= 500', 'error.contains("429")', 'error.matches("(?i)timeout|5\\d\\d")'. A predicate that doesn’t compile — or doesn’t return a bool — is rejected at save time; if a stored routine somehow carries a broken one, the runtime fails safe and does not retry rather than retrying on every error. An in-band agent failure — the agent CLI exited 0 and reported that its own turn failed — is never transient. It is the agent’s verdict on its work, not a transport fault, so a same-tier retry would repeat a deterministic failure and bill for every attempt; escalating the tier (on_fail: escalate_tier, the default) or failing fast is the useful response. The classifier decides this from the error’s identity, not its wording — a refusal that happens to read “I cannot process a list of 500 items” does not become transient because it contains 500. A retry_on predicate can still opt in explicitly with error.contains("agent reported a failed run") if a specific routine wants that. The loop is cost-aware and predictive: after each failed attempt it estimates the next attempt’s cost from the running average, and if that would push the run’s accumulated cost past max_cost_usd it stops before spending into the red — surfacing the underlying failure (wrapped, not masked) rather than a bare budget error. Each retry emits a pipeline.step.retrying event (attempt N/M) so the recovery is visible in the run trace without the run going red. on_fail: retry_step with no explicit retry: block is shorthand for the default policy: 3 attempts, exponential backoff 1s→60s, full jitter, retry any error.
Retries re-execute the step. A retried step runs its side effects again — an http POST, a script that writes a row, an agent that sends a message. Only add a retry: policy (or on_fail: retry_step) to steps that are idempotent, or scope retry_on to failure classes where the side effect provably did not land (e.g. a connection error before the request was sent). For non-idempotent work, prefer an idempotency key on the downstream API over a blind retry. When a step carries an explicit retry policy, the built-in same-tier transient retry stands down (the policy owns retries), so provider calls stay bounded at max_attempts × tiers rather than multiplying.

​
call_pipeline

Composition primitive. Save-time cycle detection rejects loops; runtime depth limit caps at 10 levels.

​
http

Egress enforcement. Every http step (and http hook) passes two host gates at run time, checked pre-flight AND on every redirect hop (CheckRedirect callback):
  1. Routine layer — egress_targets. When the routine declares egress_targets, the request host must be one of the declared hosts or a subdomain of one (api.x.com matches target x.com; evilx.com does not). A routine that declares no egress_targets is unrestricted at this layer — backward-compatible with every routine that predates the field. The SSRF guard (private/link-local IPs, DNS-rebind-safe dialing) applies regardless.
  2. Crew layer — network policy. The authoring crew’s network policy (network_mode + allowed domains — the same dial that governs the crew’s agent containers) also applies to direct http steps. A restricted crew’s routines can only reach the crew’s allowed domains (exact host match, same as the container proxy). A crew on free mode is not unconditionally open at this layer: an http step that does not declare egress_targets is still held to the same floor restricted mode enforces — the sidecar’s default LLM/CLI provider domains only. This closes an SSRF gap where any create-role member (or a webhook payload driving {{ inputs.url }}) could otherwise point an undeclared http step at an arbitrary public host just because the crew allows its agents free egress. Declaring egress_targets on the routine is the escape hatch — it bypasses this floor and the routine’s own allowlist governs instead (private/link-local IPs still blocked by the SSRF guard regardless). Webhook-triggered runs go further: they’re always held to the restricted floor, regardless of the crew’s own network_mode — the inbound payload is untrusted, so a routine fired by a webhook delivery can’t be used to reach an arbitrary host even if the authoring crew is nominally free. The crew’s own allowed_domains (if configured) still apply on top of the shared defaults, same as a genuinely-restricted crew.
A blocked request fails the step with a structured error naming the step, the host, and which layer denied it — before any bytes leave the server. Credential injection. credential_ref.type is resolved at run time against the workspace credential vault by type (case-insensitive match on the vault type, e.g. API_KEY, GENERIC_SECRET), never by ID — so a shared routine runs against any workspace holding a credential of the right type. Only ACTIVE credentials resolve; credentials pinned to another crew are invisible; when several match, the authoring crew’s own credential wins over workspace-shared ones and the newest wins within each group (rotation). The decrypted value goes into the outbound request only — never into logs, the journal, or step output. If nothing matches, the request is sent without credentials (public endpoints keep working). Injection schemes: bearer (default), header with explicit name, query with explicit name.

​
wait

Three kinds: approval (HITL token), datetime (sleep until ISO timestamp), event (external signal via POST /api/v1/workspaces/{ws}/pipeline-runs/{runId}/signal). Both approval and event survive a process restart: a top-level run parks (status: WAITING) with durable state — pipeline_waitpoints for approval, pipeline_signal_waits for event (migration v154, #1409) — and at boot the parked run is resumed from that state. For event specifically, a signal delivered while the process was down (or a hair before the step even reached the wait) is not lost: the signal endpoint records the delivery durably before attempting any in-process wake, and the resumed step reads it straight back out (see Durability and restart recovery). Async, non-blocking. Hitting a kind: approval gate does not hold a goroutine or fail the run. A foreground crewship routine run returns promptly with status: WAITING (exit 0) and a waitpoint token, and releases its execution slot while it waits. Approving or rejecting resumes the run from the gate (already-completed steps are restored/skipped); a rejection resolves the run to FAILED cleanly rather than stranding it. Parked approvals whose timeout_sec elapsed are reconciled at the next boot scan. Approve via UI Inbox, CLI crewship routine waitpoints approve <token>, or API.

​
code

A code step has a runtime. Two runtimes are wired today — expr and cel — both in-process, pure-Go, token-zero (no container, no LLM, no filesystem, no network). Use expr for a single boolean comparison (wake-gate probes); reach for cel as soon as you need real logic (booleans, arithmetic, string/list ops) — its own code comments call it out as the general-purpose deterministic primitive.

​
runtime: expr (wired, token-zero)

expr evaluates a single comparison and emits true or false:
  • Operators: > >= < <= == !=.
  • The body is Render()-ed first, so {{ inputs.x }} / {{ steps.y.output }} placeholders substitute before evaluation.
  • Anything that isn’t a single comparison (multiple operators, function calls, arbitrary code) fails closed with a clear error — expr is deliberately not a scripting language.
This is the canonical primitive for agentless probes and schedule wake-gates (emit true only when work is needed). See Wake gates.

​
runtime: cel (wired, token-zero, general logic)

cel evaluates a Google CEL expression — non-Turing-complete (every expression provably terminates), so it keeps the token-zero / no-execution-surface guarantee of expr while adding boolean operators (&&, ||, !), arithmetic, string ops, list/map membership, ternaries, and field access. Reach for it when expr’s single comparison isn’t enough. A bool result emits true/false; numeric and string results emit their canonical string form. Compile/eval errors fail closed.

​
runtime: bash | python | go (rejected at author time)

These are schema-legal runtime names but have no sandboxed runner wired. As of PR #710 they’re no longer “saves-cleanly-then-fails-at-3am”: a routine using one is rejected at save / apply / test_run time with a message pointing at the fix (runtime: expr or cel, or convert the step to agent_run).
runtime: bash, python, and go are not executable today and are rejected when you save, apply, or test-run a routine that uses them — the error names the offending step and suggests expr/cel or an agent_run conversion. Only expr and cel are wired.

​
script

The script step is the deterministic backbone of a routine: it runs a bundled script file — Python, bash, Node, any language your crew’s devcontainer installs — inside the crew’s own container, token-zero (no LLM). It’s the first-class replacement for the old workaround of an agent_run step whose prompt merely told an agent to shell out to a script: faster (no model turn), cheaper (no tokens), and reproducible (the same input always yields the same output). A routine is not an LLM chain — it’s deterministic scripts (the fast, cheap, 100%-reliable backbone) + agent steps (judgment, where the output legitimately varies). The proven pattern: a script step does the mechanical heavy lifting (parse a bank statement, reconcile, verify a checksum), and a downstream agent_run applies judgment to its output.
Then a downstream step consumes the script’s stdout as {{ steps.parse.output }}:
How it runs. The script executes in the crew container as the non-root agent user (1001:1001) via the same exec path an agent’s own shell tool uses — so it’s exactly as sandboxed as the crew already is: --cap-drop=ALL, no-new-privileges, read-only rootfs with a writable /tmp, and the container’s pid/mem/cpu caps. stdout becomes the step’s downstream output; a non-zero exit code fails the step; stderr is captured for the error/audit but never contaminates stdout. Every script step is recorded as a scrubbed exec.command journal entry (the exact argv, exit code, duration) for a durable post-hoc audit. Path + interpreter. path resolves under the crew’s shared dir (/crew/shared/); traversal or an absolute escape is rejected. interpreter is optional — it’s inferred from the file extension when omitted: An unknown extension with no explicit interpreter is an author-time error. The interpreter, path, and args are assembled into an argv (no shell), so args can never inject. Inputs flow two ways, exactly like code steps: every declared routine input is exposed as CREWSHIP_INPUT_<NAME_UPPER> in the script’s environment, and script.args / script.env values are template-substituted ({{ inputs.x }} / {{ steps.y.output }}) before the run. A default 300 s timeout applies (override with the step’s timeout_sec); stdout is capped at 1 MB. Egress. A script step runs with HTTP_PROXY/HTTPS_PROXY pointing at the crew sidecar (127.0.0.1:9119), so outbound HTTP from a script obeys the crew network policy — a restricted crew’s script can only reach hosts on its allowed_domains. NO_PROXY keeps loopback direct.
Proxy variables are reserved. Any script.env key ending in _proxy (any case) plus no_proxy is dropped and logged — a routine cannot switch off the egress fence that constrains it. The match is on the shape rather than an exact list because CPython lowercases every environment name before looking for a _proxy suffix, so HtTp_PrOxY would otherwise reach a Python script’s proxy configuration. Before this was enforced, script steps ran with no proxy at all and the crew allowlist did not apply to them (#1473).
Note this is the application-layer fence: a script that deliberately ignores the proxy (curl --noproxy '*', a raw socket) still reaches the network until the L3 fence in #1368 lands. Treat the allowlist as a guardrail for well-behaved tooling, not as containment for hostile code.

​
Deliver the script into the crew

A script step points at a file that must already live in the crew’s shared dir. Don’t hand-copy it or bake a base64 blob into the manifest — declare it on the crew and let crewship apply deliver it. Add a files: block to the crew manifest:
crewship apply materializes each file into the crew’s shared dir via the same validated /files/save path the crewship crew files CLI uses — idempotent, re-delivered when the source changes. (Ad-hoc, you can also crewship crew files save accounting shared/scripts/parse_vypis.py --file parse_vypis.py, but the manifest is the durable, declarative path.)
Overwriting a shared file needs the crew container running. A first-time delivery writes the shared tree host-side, but once a crew is provisioned its container (UID 1001) owns /crew/shared, so an overwrite of changed content — re-delivery of a modified file, or routine import --force — is routed through the running container as that UID. If the crew container is stopped, that overwrite returns 409 with a “start the crew and retry” message rather than a silent failure. Re-delivering identical content is always a no-op (it succeeds even on a stopped crew), so a steady-state crewship apply never needs the container running.
A script is a crew asset, not a routine field — it lives in the crew’s shared dir and is shared by every routine that crew authors. The crew manifest files: block is the source of truth. See below for how a script still travels with routine export/import.The files: block works in both crew manifest shapes — a standalone kind: Crew document and a combined manifest that also nests agents:/skills:. Either delivers the file into /crew/shared at apply through the same validated path.
Per-step egress is NOT enforced at exec. A script step (like an agent’s shell) inherits the crew container’s global network policy; only http steps honor a step’s egress_targets. Don’t rely on a script step for per-step network isolation — put outbound calls that need an allowlist in an http step.

​
Scaffold and portability

  • Scaffold a script-backed routine with crewship routine init --script — it emits a script step feeding a downstream agent_run, with delivery instructions in the description.
  • Portable bundles. crewship routine export inlines every referenced script file (base64) from the routine’s author crew, so a portable routine travels with its backbone — recipe + scripts + agent judgment. crewship routine import --crew <slug> re-materializes the inlined scripts into the target crew (same /files/save path). A script whose destination already exists with different content fails loudly — pass --force to overwrite (remember: a crew script is shared across routines, so overwriting may affect another routine). --no-scripts on either command skips inlining/materializing. Even with inlining, the crew manifest remains the source of truth; export/import is a portability convenience.

​
transform

Pure-Go data reshaping with a tiny jq-flavoured subset. No LLM, no network — fully deterministic. Useful for wiring step outputs without paying for another agent_run.

​
notify

A non-blocking inbox message pushed mid-run — the push, author-controlled complement to wait: approval (which blocks for a decision). notify posts a rendered update and continues, so you can drop progress pings anywhere in the DAG (“extracted 3 invoices”) and a final result at the end. No LLM, no network. Saying what the notice is. Without category, every notify step routes as chat.replies — the category an inbox message maps to. That is fine for a progress ping and wrong for everything else: “the nightly deploy failed” and “here is your weekly digest” arrive under the same label, and the preference matrix people tune is invisible to you, the one author who knows what the event actually is.
One announcement, not two. Crewship also emits a routines.completed notification of its own when a run finishes, so a routine that says nothing still reports that it ended. If your routine unconditionally tells the whole workspace it finished under that same category, the generic one is suppressed — your message says what happened and carries the result, and getting both every run is how a channel earns a mute. Suppression is deliberately narrow: a notice behind an if, or aimed at a role, a user, a crew or the trigger, reaches fewer people than the generic one, so that one still goes out. A category is a routing label, not a privilege: it selects which row of a recipient’s matrix the notice is matched against, and every row is still opt-in per user and still gated by the channel’s allowlist. An unknown category is rejected when the routine is saved — routing into one nothing matches would deliver to nobody while the run reported success. The full vocabulary is in crewship notify prefs --help. Targeting a crew. to: "crew:<slug>" addresses a crew’s human audience — everyone listed as a member of that crew — and delivers one inbox card per member:
The slug is resolved inside the run’s own workspace (crew slugs are unique per workspace), so a slug can only ever address crews in the same workspace as the run. Each member’s card gets its own idempotency key — (run, step, user) — so a resumed or retried run never double-posts, and the anti-spam soft cap below applies per member independently. Every member is notified; there is no role filter or opt-in subscription yet. If the crew doesn’t exist in this workspace, has no members, or can’t be looked up, the notice degrades to a workspace-wide notice rather than vanishing or failing the run: the step output is marked notified:degraded and the run carries a warning, so a typo’d slug is visible instead of silent. The message lands live (WebSocket broadcast to members) and scrubbed (secrets in mid-run outputs are redacted before insert). Delivery is best-effort: a routine whose notification can’t be written never fails on that account — the run’s real work is already done. Retries are idempotent — each notify is keyed on (run, step) ((run, step, user) for a crew fan-out), so a re-run doesn’t double-post. Routine updates carry a routine_update subkind so the inbox can keep them in their own lane, separate from approvals and escalations. Anti-spam soft cap. A single run may deliver at most 20 notices to any one recipient — enough for progress + result pings, low enough that a routine dropping a notify into every step of a wide DAG can’t bury someone’s inbox. It’s a soft cap: once a recipient hits it, further notices to them in that run are quietly dropped (logged), never a run failure. The cap is per (run, recipient), so notifying several people each gets its own budget, and a schedule firing repeatedly is unaffected — each run is a fresh budget. For the zero-config version — a desktop ping when a run ends (success or failure) without authoring a step — run it with crewship routine run <slug> --wait and enable notifications (crewship notify enable). To reach outside Crewship when a run completes or fails — email or a signed webhook to an admin, a Slack relay, or your own service — configure outbound notification channels. The notify step is the in-product half; outbound channels are the half that reaches people who aren’t watching the inbox.

​
query

A deterministic, read-only aggregate query over the run’s own workspace’s operational data — no LLM, no network egress, agentless-compatible. source: "pipeline_runs" is the only source today: run counts, cost, and top failures over a trailing window (default 24h, capped at 720h/30 days regardless of what’s authored). This is the data source behind the workspace digest routine template. Output is a JSON object — pipe it through a transform step to pull a single field for a downstream notify/http step, exactly like any other structured step output:
{ window_hours, total_runs, completed, failed, waiting, cancelled, total_cost_usd, top_failures: [{pipeline_slug, count}], summary_md } — summary_md is a pre-rendered markdown digest, so the common case (query → notify) needs only the one transform step shown above to extract it. Strictly workspace-scoped: the query always runs against the invoking run’s own workspace_id — a routine can never read another tenant’s run history no matter what it’s authored to ask for.

​
foreach — fan out over an array

Runs the body once per element of a JSON array, collecting each item’s result into a JSON array as the step’s output ({{ steps.process_each.output }} → ["…","…"], in input order). Within an item the body runs like a mini-pipeline: per-step retry:, validation, outcomes, and tier escalation all apply. Cost is summed across every item and attributed to the foreach step, so max_cost_usd still bounds the whole fan-out. The first item that fails cancels the rest and fails the step (fail-fast). wait, call_pipeline, and a nested foreach are not allowed inside the body — the fan-out is a bounded, self-contained unit.

​
Conditional if

Any step can carry an if: gate. Two forms, chosen automatically:
  • CEL expression (no {{ … }}): evaluated against the typed variables inputs, steps, and run. Real comparisons and logic work — "if": "inputs.dry_run == false", "if": "steps.classify == \"spam\"", "if": "inputs.count > 0 && run.is_replay == false". A boolean result gates directly; a number/string result falls through to the truthy rule below.
  • Template ("if": "{{ inputs.run_summary }}"): rendered to a plain string, then truthy-checked — empty / false / 0 / no / off (case-insensitive) → step skipped and marked <skipped> in StepOutputs; anything else is truthy. A bare literal like "yes" / "false" follows the same truthy rule.
The CEL variables are: inputs.* (the run’s inputs, numbers stay numbers for arithmetic), steps.<id> (that step’s output string), and run.metadata.* / run.is_replay / run.replay_of. A skipped step never trips on_fail — “didn’t run” is structurally different from “ran and failed”.

​
DAG with needs[]

Steps with no overlapping needs execute in parallel (one goroutine wave per ready set, bounded so a wide fan-out can’t stampede the provider into rate limits). Final output picks the unique leaf node; for multi-leaf DAGs the first leaf in source order wins.

​
parallelism — auto-derive the DAG

By default a routine is sequential unless you wire needs: (GitHub-Actions parity: no needs = depends on the previous step). Set a routine-level parallelism to change how independent steps are scheduled:
The three summaries reference nothing upstream, so under auto they run in one wave (~1× wall-clock) instead of one-after-another — no needs: required. auto is opt-in because it changes the implicit “depends on previous” semantics; a routine you didn’t mark keeps running exactly as before.

​
Lifecycle hooks (before_all / after_all / on_failure)

Routine-level hooks run deterministic side-channel steps around the main execution — a clean home for setup/teardown that isn’t part of the visible step graph. Hook steps must be code / http / transform (no agent_run — a hook must not recurse or spend tokens):
after_all and on_failure are best-effort — logged, but they never change the run’s outcome. Hooks fire only on the top-level run (not nested call_pipeline expansions) and are skipped on resume re-entry and dry-run. Per-step before / after hooks also exist and are included in the capability manifest walk. Full reference: Lifecycle hooks.

​
Per-step overrides (no version bump)

Tweak a single step’s prompt or model without bumping the routine version — the override is applied at run start over the versioned DSL, so the durable, authored definition stays the source of truth while an operator can patch and clear a live behavior quickly:
Only non-empty fields win — a prompt-only override leaves the authored model untouched. Full reference: Per-step prompt/model override.

​
Debug a single step (step-run)

Iterating on one step’s prompt shouldn’t cost a whole pipeline run. crewship routine step-run executes a single step against an input fixture, in isolation — no upstream steps, no DAG, no persisted run record — and prints the step output, validation verdict, and (for agent_run) cost. Supports agent_run, http, script, and transform. The deterministic three used to have no “run just this step” option at all — you had to run the whole pipeline to see whether a transform expression or an http step’s URL rendered correctly, even though they’re the cheapest, most deterministic steps in the DSL (no LLM, no waiting):
  • --input is the step’s inputs fixture — a JSON object, inline or @file.json. The step’s {{ inputs.X }} refs resolve against it.
  • --outputs seeds upstream {{ steps.X.output }} refs — a JSON object mapping step_id → that step’s output (a string, or any JSON value, which is stringified). Most non-first steps read an upstream output, so without --outputs those refs render empty and you’d be iterating against different input than a real run — so the command warns loudly (⚠ step references {{ steps.parse.output }} but no --outputs fixture was provided…) when a referenced upstream output isn’t seeded, across whichever fields the step type actually templates (prompt for agent_run; url/body/headers for http; input/expression for transform; args/env for script). Still no DAG traversal — the fixture stands in for upstream steps; nothing upstream executes.
  • --tier-override (trivial|fast|moderate|smart) swaps the step’s tier for cheap structural iteration on an agent_run step; a step-level model_override still wins. Ignored for http/script/transform — there’s no tier to override.
  • The response is marked simulated: true — it does not create a run record, so it never shows up in routine records/metrics or pollutes the run trace. A script step-run still executes for real in the crew’s container (real side effects — only the run-record bookkeeping is skipped), and an http step-run still goes through the same egress/SSRF guard a real run enforces, not a relaxed check.
  • Not supported: wait (blocks on external state), notify/call_pipeline (side effects that don’t make sense without a real run), and code (needs the same sandboxed-container wiring script has, not yet extended here).
This is the “unit test for one step”: seconds instead of the full pipeline. It complements the two tools that leave a gap — routine dry-run renders prompts but executes nothing, and a full run executes everything but is slow and costs real tokens. Backed by POST /api/v1/workspaces/{ws}/pipelines/{slug}/step_run.

​
Agentless routines

Declare "agentless": true to get a token-zero guarantee: the routine can never invoke an LLM, no matter who edits it later. This is what makes high-frequency automation (health checks, metric probes, TLS expiry watches) free to run on a tight cron.
The probe’s verdict comes from data reshaping plus a single comparison, not scripting: the http step fetches a JSON status, transform projects the number out of it, and the expr code step emits the boolean true/false — all token-zero. (A pure http + transform projection of an already-boolean field works too; reach for expr when you need the comparison.) Enforced at two layers:
  • Save time — validation rejects agent_run (direct LLM spend), call_pipeline (the target resolves by slug at runtime, so a nested routine could gain an agent step later and silently break the guarantee), and eval.online with sample_rate > 0 (online grading runs a grader agent against the routine’s runs).
  • Run time — the executor independently refuses to dispatch an LLM-capable step inside an agentless run, so even a definition written before the validator existed fails closed.
Everything else works as usual: egress allowlist, credentials, versioning, dry-run, schedules, webhooks. Agentless routines are also the only valid probes for wake gates.

​
Two-tier execution

The economic value-prop: an Opus-class authoring model designs the routine, a Haiku-class executor model runs each invocation. Workspace execution_tiers_json maps complexity classes to (adapter, model):
Per-step complexity annotation drives the resolver. With on_fail: "escalate_tier", a failed validation walks the fallback chain — practically: Haiku tries first, Sonnet on validation fail, Opus on second fail.
Tier override at runtime. The CLI flag --model <model> is constructed from the resolved tier and passed to the agent’s CLI adapter, so a routine’s complexity: "fast" actually fires Haiku, not the agent’s default. CLIAdapter is preserved (so the agent’s CLAUDE_CODE / GEMINI_CLI / etc. wiring stays intact); only the model name swaps.

​
Save validation gate

Save endpoints (sidecar /pipelines/save, user /api/v1/workspaces/{ws}/pipelines/save, internal /api/v1/internal/pipelines/save) require the routine to clear a validation gate before it persists. The gate is a dry-run validation of the draft, not a real execution — there is no “test run” mode (you cannot run an agent dry). The sidecar agent-authoring flow forwards the draft to /api/v1/internal/pipelines/test_run, which parses, schema-validates, and dry-runs it (rendering every template, invoking no agent); on success it mints an HMAC save_token bound to (workspace, definition hash, authoring crew) and forwards that token to save. InternalSave verifies the token — it does not trust a body-supplied “it passed” claim.
Autonomous ≥ interactive: the gate never weakens for agents. The agent-authored path (InternalSave) is held to the same proof-of-test bar as the interactive user path. Both clear the store gate only against a server-verified save_token minted by a real dry-run — the old last_test_run_at / last_test_run_passed body fields are forgeable and no longer trusted on either path (#1371). An agent cannot author and activate a routine that never passed a dry-run by simply claiming it did. This is a security invariant: every gate that guards an interactive save must guard the unattended one at least as tightly.
This is the self-improvement loop: an authoring agent that writes brittle DSL gets a structured failure report it can read and revise from. Without the gate, MVP would ship pipelines that pass schema but fail at runtime. Real execution happens on the first live run, and risky routines are human-reviewed (governance) before they go live.
skip_test_gate: true is honored only when the caller’s role is OWNER or ADMIN; lower roles get 403. Useful for hand-crafted DSL from known-good templates (the seed flow uses it).

​
Governance — agent proposes, human approves the risky ones

Routines have a lifecycle status (migration v128): active (live + runnable), proposed (awaiting approval), or disabled (admin airbag). The save validation gate still applies on top — status is an additional gate.

​
Maker-checker on save

When a routine is saved (by an agent via the sidecar, or by a user via the UI/CLI), Crewship classifies it. A save is risky if any of these hold:
  • it declares an integrations_required the author crew can’t currently satisfy (the same resolver the run gate uses);
  • it has any http/egress step (or routine-level egress_targets);
  • it has any code-runtime step;
  • it declares credentials_required.
Otherwise it’s safe — only agent_run / transform / call_pipeline / wait steps over satisfiable integrations, no egress, no credentials.
  • Safe → active. Goes live immediately, exactly as before.
  • Risky → proposed. The routine is persisted but not runnable, and a blocking inbox item is raised for MANAGER+ (the same Inbox surface as proposed skills). Approve it to go live, or reject it.
A proposed (or disabled) routine refuses run / run_batch with 409 Conflict — "routine is awaiting approval" or "routine is disabled". dry_run always previews a saved routine, so it’s never blocked.
OWNER/ADMIN escape hatch — skip_governance_gate. Symmetric with skip_test_gate: passing "skip_governance_gate": true on the user save (POST /api/v1/workspaces/{ws}/pipelines/save) forces a risky definition live as active and raises no review item. Honored only for OWNER/ADMIN (lower roles get 403); it is deliberately not available on the agent/sidecar save path (InternalSave), so agent-authored risky routines are always reviewed. This is what the crewship seed flow uses so a freshly-seeded workspace’s hand-curated starter routines are immediately runnable instead of stuck “awaiting approval”. Use it only for DSL you trust.

​
Approve / reject (MANAGER+)

  • POST /api/v1/workspaces/{ws}/pipelines/{slug}/approve — MANAGER+. Flips to active.
  • POST /api/v1/workspaces/{ws}/pipelines/{slug}/reject — MANAGER+. Soft-deletes the proposed routine.

​
Disable / enable (OWNER/ADMIN airbag)

  • POST /api/v1/workspaces/{ws}/pipelines/{slug}/disable — OWNER/ADMIN. Flips to disabled and cancels any in-flight runs of that routine immediately.
  • POST /api/v1/workspaces/{ws}/pipelines/{slug}/enable — OWNER/ADMIN. Returns it to active.
List responses (and crewship routine list) carry status; filter the queue with:

​
Triggers

​
Cron schedules

5-field cron expression. Scheduler runs in-process and ticks every 30s, so minimum resolution is 1 minute. Multiple schedules due in the same tick fire concurrently (a bounded pool of up to 12 at once) — a slow routine no longer stalls every other co-due schedule for the rest of that tick.
Single-instance only — running multiple replicas would double-fire (no leader election yet).

​
Natural-language schedules (--when)

Pass --when instead of --cron for the common phrasings — it parses to a cron expression and echoes the next 3 fire times so you can confirm the schedule means what you intended before it saves:
Supported phrasings: every day at 9am, every weekday at 9, every weekend at 10am, every monday at 14:00 (any named weekday), every hour, every 15 minutes, every 2 hours. --cron and --when are mutually exclusive; --when respects --timezone for the preview. Pass --yes to skip the confirmation prompt (scripts / CI). Anything outside that phrase set returns an error asking for a raw --cron expression instead of guessing wrong.

​
Missed-run catch-up

If a schedule falls overdue by more than one cron occurrence — the server was down, the schedule sat disabled for a while, next_run_at otherwise fell behind — catchup_policy decides what happens to the backlog:
Whenever a tick drops or collapses backlog occurrences, routine schedules list shows it in the MISSED column (4 (skip), 3 (once)) instead of the gap being silently invisible, and a medium-priority MANAGER inbox notice is raised the first time it happens for that occurrence. catchup=all never “misses” anything (MISSED stays —) since every occurrence in the backlog actually fires.
A wake-gated schedule (see below) always behaves like catchup=once — the probe is evaluated live (“is now the moment to wake”), never re-run against a backdated historical timestamp, so skip/all only apply to plain schedules. Missed occurrences are still counted and surfaced for a gated schedule; only the multi-fire/no-fire policy expansion doesn’t apply.

​
Wake gates

A plain cron fires the full routine — including its LLM steps — on every tick, even when there is nothing worth the model’s attention. A wake gate fixes that: the schedule references an agentless probe routine, the scheduler runs the probe first on each tick (free of LLM spend by the agentless guarantee), and the main routine fires only when the probe’s final output is truthy. Same falsy rule as step if: conditions — empty, false, 0, null, nil, no, off (case-insensitive) skip the tick; anything else wakes the routine.
This gives schedules a cost ladder: agentless schedule (always free) → wake-gated schedule (probe free, LLM only on signal) → plain schedule (today’s default, unchanged). A freshly-seeded workspace ships the routines for this pattern unscheduled: feed-watch-probe (agentless) and feed-change-report (agent routine). Wire them together yourself to see it work:
Point the probe’s url/expected_items inputs at your own endpoint to make it real. Semantics worth knowing:
  • The probe must be agentless: true, live in the same workspace, and can’t be the schedule’s own routine — all validated when the schedule is saved.
  • Probe errors fail open by default: a broken or deleted probe wakes the main routine instead of going silently blind, and records last_wake_status: ERROR so you can see the probe needs fixing. This is the right default for a monitoring gate — you would rather over-fire than miss a signal.
  • Fail closed for unattended gates: when the gate is meant to suppress an autonomous run — not just save tokens — a broken probe that fires the run anyway is a security hole (a tampered or unreachable probe cannot be allowed to green-light the run). Pass --fail-closed so any non-affirmative probe outcome (error, timeout, or a non-COMPLETED run) holds the run instead of firing it. The tick records last_wake_status: HELD, does not increment wake_fire_count, and advances next_run_at like a skipped tick. The default stays fail-open, so existing schedules are unchanged.
    Note an empty or ambiguous output on a COMPLETED probe run is already treated as falsy → SKIPPED (the main run never fires) regardless of policy — --fail-closed only changes the failure branch.
  • A skipped tick advances next_run_at but leaves last_run_* untouched — run telemetry stays strictly about main runs. Wake telemetry lives in wake_check_count / wake_fire_count / last_wake_at / last_wake_status, and routine schedules list shows it as <probe> woke/checked in the WAKE column.
  • Probe executions are regular runs with triggered_via: wake_check, so they’re auditable in run history and filterable out of dashboards.

​
Circuit breaker

A schedule whose target routine is broken (deleted agent, expired credential, a bug in the routine) fires on every tick, fails every time, and each failed run raises a MANAGER inbox alert — unbounded inbox spam and burned agent cost with no backstop. The circuit breaker auto-disables a schedule after 5 consecutive failed fires (the default; override per schedule with --max-failures), so a broken cron goes quiet instead of failing forever.
When the breaker trips:
  • The schedule is disabled (enabled: false) with disabled_reason: circuit_breaker — distinct from an operator-initiated disable (disabled_reason stays empty), so routine schedules list and routine doctor can tell you why a schedule went dark.
  • A journal event (pipeline.schedule.circuit_breaker_tripped) and exactly one actionable MANAGER inbox alert fire for the trip itself — separate from the per-run failed_run alerts, which still fire once per failed run.
  • A successful fire resets the consecutive-failure counter to 0, so an intermittently-flaky routine that eventually succeeds never trips.
routine schedules list surfaces the live counter in the FAILS column (2/5) and the tripped state inline in ENABLED (no (circuit_breaker)); routine doctor <slug> reports the same state as a schedule_circuit_breaker check (OK / WARN while approaching the threshold / FAIL once tripped). Re-enabling clears the slate — the counter resets to 0 and disabled_reason clears:

​
Missed occurrences on recovery

next_run_at is always computed from time.Now() at fire time, so downtime spanning several cron occurrences (server down for 3 hours on an hourly schedule) yields at most one fire on recovery — the routine still only runs once, it does not backfill the missed occurrences. Before #1409 that gap was invisible; now the scheduler emits one pipeline.schedule.missed_occurrences journal event per schedule when it detects the gap, reporting how many occurrences were skipped and the time window: “schedule X skipped 11 occurrence(s) between <window_start> and <window_end>”. This is observability only — pair it with the circuit breaker and journal search if you need to know a schedule went dark for a while.

​
Webhooks

Output reveals the public URL and signing secret once (Stripe-style). External services POST event payload to /api/v1/webhooks/{token}. With HMAC, sender includes header X-Crewship-Signature: sha256=<hex_hmac_of_body>, validated server-side via hmac.Equal (timing-safe). Rate limited per token, per minute, default 60. Replay defense — timestamped signatures (optional). A body-only HMAC is replayable for as long as the delivery’s idempotency reservation is live (up to 24h, and a failed run can reopen the window). To close that gap, a sender can additionally include X-Crewship-Timestamp: <unix_seconds> and sign "<timestamp>.<body>" instead of the bare body:
When X-Crewship-Timestamp is present, the signature must cover the timestamped material AND the timestamp must be within a 5-minute freshness window — a captured signed request becomes useless once that window passes, even if it’s replayed before the idempotency reservation expires. Omitting the header falls back to the body-only scheme unchanged, so existing senders keep working without any change. Injection fence + forced egress for webhook-triggered runs. The request body lands in the routine’s inputs as inputs.event (parsed JSON or raw string), inputs.raw (the raw bytes as a string), and inputs.headers — all attacker-influenced if the upstream sender’s payload carries untrusted text (an issue title, a commit message, a customer-supplied field). Two protections apply automatically, with no DSL changes required:
  • Any agent_run step’s rendered prompt wraps those three input values in the same untrusted-ingress fence (<untrusted source="webhook" ...>) the agent-webhook path already uses — the model is told to treat fenced content as data, never instructions. Other template destinations (http URL/body/headers, code, transform, if) are not fenced — wrapping those would corrupt a URL or a JSON body rather than protect anything; they’re covered by the egress hardening below instead.
  • Every http step in a webhook-triggered run is held to the crew network policy’s restricted floor regardless of the crew’s actual network_mode — see Egress enforcement above.
Delivery is asynchronous. The endpoint verifies the signature, rate limit, governance status, and the routine’s concurrency_key gate synchronously, reserves a run id, then answers immediately:
The routine executes in the background under the returned run_id, so senders with short delivery timeouts (GitHub ~10s, Stripe ~5s) never time out on long agent runs — and a sender closing the connection early cannot cancel an in-flight run: the run’s context derives from the server lifecycle, not the HTTP request. Poll the handle for the outcome:
routine result answers “what did run X produce?” — it re-fetches the run and prints the final output (structured JSON is pretty-printed; --format json for scripting), followed by a Files changed during this run section: the files on the run’s crew written inside the run window (report.pdf, summary.md, …) with a crew files get fetch hint. For a document-producing routine the files are the deliverable. Read the list as a correlation (“what changed on the crew while this ran”), not proof of authorship — a concurrent session or overlapping run on the same crew shows up too; see the CLI reference for the full caveat. routine logs now includes the output too; use result when the deliverable is all you want.

​
Client-facing views

The commands above are operator-shaped (run ids, tiers, cost columns, entry_type labels). Three affordances turn a run into something you can put in front of a non-engineer:
  • Live progress — crewship routine watch <slug> --progress collapses the raw event tail into one updating status line: ▶ reconcile-invoices · step 2/4 (verify) · RUNNING · $0.0021 · 12s. Step count comes from the routine definition; cost + elapsed from the run’s events. It exits when the run finishes. (Without --progress, watch stays the ANSI event tail for CI/scripting.)
  • Shareable report — crewship routine report <run_id> assembles a run into a readable document: inputs → each step’s outcome + output → the final deliverable → cost & duration. Markdown by default (paste into a ticket/chat); -f html -o run.html writes a self-contained page to hand to a customer (output is HTML-escaped — no injection from run data).
  • Redacted mode — --client on result, report, and logs drops the internal noise (run-id, cost, tier, mode, trigger, entry_type labels) and renders status in plain words (Succeeded / Failed), leaving just the routine name, inputs, and deliverable.
Redelivered events dedupe synchronously: a replay (same Idempotency-Key / X-Crewship-Event-ID, or identical bytes within the dedupe window) answers 202 with "status": "DEDUPED" and the original run’s id — the routine executes exactly once. A proposed/disabled routine answers 409 (policy block, nothing dispatched). A delivery arriving while the routine’s concurrency_key gate is at capacity answers 429 with a Retry-After header before anything is dispatched — a 429 never consumes the idempotency key, so redelivering the same event later executes it normally. Runs that hit a wait: approval step park as WAITING and resume once the waitpoint is approved, exactly as with other triggers.
The signing secret is shown once. To rotate it: delete the webhook + create a new one. There is no in-place rotation by design.

​
Manual

crewship routine run <slug> --inputs '{...}' or click the Run button in the UI detail panel. Same execution path. After you click Run or Test run in the UI, a live Run activity rail appears inline in the detail panel showing the just-started run step by step (started → each step → completed/failed) — so status is visible immediately without switching to the Runs tab. Full run history stays in the Runs tab; see the Activity guide for the rail, and the toolbar Activity Bar for a workspace-wide “what’s running now” view.
The Test run button calls the public, JWT-authed test_run endpoint (POST /api/v1/workspaces/{workspaceId}/pipelines/test_run). It validates a draft — parse + Validate + the integration and resource preconditions + a dry_run pass (no agent is invoked; you can’t run an agent “dry”) — and on success mints an HMAC save_token bound to (workspace, definition hash, user). Save verifies that token, so a draft can’t be saved as “test passed” unless it actually passed test_run. The UI button and the CLI both use this endpoint. To preview a saved routine instead, use Dry run — it walks the saved definition, renders templates, and returns the declared manifest (the blast radius) without invoking anything.

​
Deferred dispatch: delay, ttl, debounce, priority

A triggered run can be parked instead of firing immediately — useful for “run 60s from now” scheduling or for coalescing a burst of near-duplicate triggers into one run:
An in-process dispatcher (5s tick) fires due rows highest-priority-first and expires rows past their ttl. Immediate runs (no --delay / --debounce-key) are unaffected. Full reference, including the underlying API fields: Deferred dispatch.

​
Dry-run preview

Two execution modes, distinguished by Mode in the request body and surface: There is no test_run mode: you cannot run an agent “dry” — it executes arbitrary scripts (bash, ansible, curl) whose side effects can’t be intercepted — so a real run is always run. The agent-authoring save gate validates a draft via a dry_run (structure + templates), not a real execution. Dry-run is the safe “what would this routine do?” preview, and an honest static plan — not a proof the run will succeed. It walks the DSL, renders all template substitutions against the supplied inputs, resolves each step’s execution tier (adapter + model), and reports a would_execute list with per-step estimated cost. It also returns the routine’s declared manifest — the full blast radius (integrations, egress, credentials, agents, routines, datastores, tools, has_http, has_code) — so the UI can show “would use: ansible, Postgres, discord.com, agent jordan”. (A definition that no longer parses leaves manifest null and still returns the report.) No agents are invoked; no journal entries beyond a single pipeline.dry_run audit row are written.
In the UI: click Dry run in the routine detail panel. The would_execute report renders inline above the tab bar with per-step:
  • Step ID + type
  • Resolved tier_adapter:tier_model (e.g. claude:claude-haiku-4-5)
  • Estimated cost in USD (order-of-magnitude only — the executor uses a flat token-density heuristic, not real pricing)
  • would_call_agent / would_call_pipeline target
The estimate is intentionally labelled “estimated” everywhere it surfaces — it’s a planning aid, not a quote. Real cost only lands once you switch to run.

​
Versioning + rollback

Every save creates a new immutable row in pipeline_versions (v79 migration). The pipelines.head_version column points at the current. Rollback repoints HEAD at the target version and makes its definition live — no new row is created (versions are deduped by content hash), and routine versions marks the new HEAD:
History is preserved — you can roll forward to a later version by another rollback. There is no “delete version” — if a version was bad, the trail stays: “v1 → v2 → v3, HEAD back on v2, then v4 (fix)” is the audit story you keep, and each run additionally records the content hash it executed.

​
Diffing versions

crewship routine diff <slug> --from N --to M prints a unified diff between two versions’ definitions (pretty-printed JSON, so a single field change shows as a single-hunk diff rather than the whole minified blob changing):
rollback calls the same endpoint automatically and prints a “What changed” view against the version it just rolled back from, so you see exactly what the rollback undid without a separate command. Identical versions (same definition_hash) print a one-line confirmation instead of an empty diff. See the CLI reference for the JSON shape and --format json output.

​
The improvement loop: routine iterate

Version history is also the substrate for self-improving routines. crewship routine iterate closes the loop that ad-hoc “agent loop” setups hand-roll in a single chat session — run, score, rewrite, repeat — but with the runtime’s guarantees attached:
Each round: the routine runs with your inputs → a grader agent scores the output 0–100 against your rubric → below --target, an optimizer agent proposes an improved definition → local validation → you confirm → the definition saves as a new immutable version. The change_summary on each version records the provenance (iterate round 2: score 74/100 — misses the TL;DR), so routine versions and the versions panel read as the improvement journal — and rollback undoes any round. What makes this different from looping in a chat window:
  • Budgets are enforced, not advised — every run and agent call lands in the Paymaster ledger under workspace budgets.
  • The gates don’t move — each save passes the same server-side test-run gate and governance rules as a human author; a definition that gains risky steps (http, code, egress) parks as proposed for review and the loop stops.
  • Nothing is lost — full journal per run, one version row per accepted round, reversible by design.
The rubric is the contract; write it like a brief to a colleague. Grader and optimizer are ordinary crew agents — a dedicated “reviewer” persona works well as the grader. Full flag reference in the CLI docs. Housekeeping is automatic: the one-shot grader/optimizer chat sessions are deleted after each call (they’d otherwise pile up in the agent’s session sidebar), and if a later round errors mid-loop the summary table still prints for the rounds already scored — you never lose the scores you paid for.

​
Bundle export / import

Routines are portable across workspaces:
--crew is required on import: it names the author crew that owns the routine and resolves its agent slugs. Scripts inlined in the bundle materialize into that crew’s shared dir (pass --force to overwrite a differing existing script, or --no-scripts to skip materialization). The bundle format is crewship-pipeline-bundle/v1: routine row + (optionally) the full version chain + change_summary annotations. Author identity is rewritten on import so the importing user becomes the new author. Slug is preserved; if it conflicts in the destination workspace the existing row updates (new version), or you change the bundle’s slug before import.

​
HITL waitpoints

A routine that includes a wait step of kind: approval parks the run on a DB-backed waitpoint — without holding a goroutine or an execution slot. The triggering crewship routine run returns immediately with status: WAITING and the token (see the wait step). Operators decide via three paths:
When you trigger a run from the routine detail page and it parks on an approval gate, you don’t have to leave the page: the Run activity panel switches its status to amber “Waiting for approval”, pins the parked step in the timeline, and shows an inline Approve / Reject banner (with an optional comment) for that run’s waitpoint. The same pending item also surfaces in the top-bar Inbox bell and the left-sidebar Inbox badge, so an approval waiting on you is visible whether or not you’re looking at the routine. The workspace-wide Wait points tab and /inbox remain the places to act on approvals for runs you didn’t just start. The decision comment is forwarded to the parked run as the wait step’s output, so downstream steps can read approval rationale via {{ steps.<wait_step_id>.output }}.

​
Durability and restart recovery

Run state is persisted to the pipeline_runs table at every step boundary: when a step starts, current_step_id is stamped; when it completes, the full step-outputs map and accumulated cost are flushed. A hard kill (crash, OOM, kill -9) therefore loses at most the step that was in flight. At boot, the server scans for runs left in queued/running from the previous process lifetime and resumes them from the next unfinished step:
  • Completed steps are restored, not re-executed. Their outputs feed downstream {{ steps.X.output }} templates exactly as if the process had never died.
  • The in-flight step re-executes from scratch — at-least-once semantics. For an agent_run step this means the agent call is re-issued (and re-billed); http/code/script steps with external side effects may fire twice. Design steps to be idempotent where that matters.
  • Runs parked on a wait approval step re-attach to the original waitpoint token. No duplicate approval card is created; the pending inbox item stays answerable across the restart, and approving it resumes the run.
  • Runs parked on a wait: event step re-attach the same way (#1409): the wait durably ARMs a pipeline_signal_waits row before parking, so a signal delivered via POST .../signal at any point — including while the process was down — lands in that row rather than an in-memory-only channel. Resume checks for an already-delivered payload first; if none has arrived yet it re-registers and waits again.
  • DAG runs (needs:) resume at wave granularity — the parallel scheduler flushes state when each wave completes, so a kill mid-wave re-executes that wave’s unfinished steps.
  • call_pipeline boundaries are NOT persisted. A kill while a nested pipeline is executing re-runs the entire nested pipeline on resume — the parent’s call_pipeline step is the unit of recovery, and the nested run’s own per-step progress is not checkpointed. Keep nested pipelines short or idempotent if a mid-flight kill matters to you.
  • The accumulated cost is restored too, so max_cost_usd keeps counting across the restart instead of resetting. Caveat: cost is flushed at step boundaries, so whatever the killed in-flight step had already spent before the kill is not in the restored total — the cap under-counts the true spend by up to one step’s worth (and the re-executed step is billed again on top).
  • A resumed run that finds its concurrency slot occupied waits and retries with capped exponential backoff (2s doubling up to 60s) instead of failing — losing the slot race to a freshly-fired scheduled run is a timing collision, not a reason to abandon hours of restored work. If the server shuts down while a run is still waiting for its slot, the row stays in-flight and the next boot resumes it again.
  • A waitpoint that timed out while the process was down resumes, observes the expired token, and fails with wait step "X" (approval) timed out — distinct from an operator clicking deny (… denied).
When persisted state is insufficient to resume safely, the run is stamped interrupted instead — never silently dropped, never wrongly resumed. Fallback triggers: the pipeline row is gone or no longer parses, the definition changed since the run started (content-hash mismatch — this catches in-place edits even when every step id survives, not just renamed/removed steps), unreadable persisted inputs/outputs, or a non-resumable mode (only live run rows resume; a dry_run preview row from a previous lifetime is never re-run). The reason lands in the run’s error_message. Graceful shutdowns are different: an in-flight run cancelled by shutdown is finalized as cancelled (a terminal state) and is not resumed at next boot. Resume targets hard kills, where no terminal write could land. Set CREWSHIP_PIPELINE_RESUME=off to disable resume and restore the older stamp-everything-interrupted behaviour — useful if a crash loop would otherwise re-burn the in-flight agent step’s tokens on every restart. Default is on.

​
Online eval sampler

The online sampler watches completed routine runs and grades a configurable percentage of them through the existing rubric grader so production traffic continuously feeds the drift detector — not just on-demand replays or scheduled regression suites. Per-routine DSL:
What happens on a tick (default cadence: every 1 minute):
  1. Scan pipeline_runs WHERE status = 'completed' AND completed_at > watermark.
  2. For each candidate, resolve the routine DSL. If eval.online is absent or sample_rate <= 0, skip.
  3. Draw from crypto/rand. If the sample lands above sample_rate, skip.
  4. Otherwise, INSERT into eval_runs with kind = 'online', status = 'queued'. The existing grader worker picks it up and writes the result back.
Correctness guarantees:
  • Schema-layer idempotency — eval_runs has a partial UNIQUE INDEX (pipeline_run_id) WHERE kind='online'. A duplicate sampler instance or a crash-recovery watermark replay can attempt the same enqueue twice; the second collapses to a no-op rather than queueing twice and double-billing the grader.
  • (completed_at, id) tuple cursor — parallel fan-out steps that complete at the same nanosecond all get graded; a timestamp-only cursor would orphan siblings.
  • Stuck-on-error watermark — a transient per-row error (resolver outage, entropy outage, enqueue conflict) freezes the watermark at the row before the failure so the next tick retries. Deterministic skips (no eval config, sample roll missed) advance normally.
  • Trace correlation — the eval row carries routine_slug + pipeline_run_id so an operator clicking a low-scoring grade in the eval UI lands on the actual trace via pipeline_run_id -> trace.
Limitations:
  • Watermark is in-memory only. On process restart it resets to now - 1h; outages longer than that leave a gap in grading coverage. The UNIQUE index keeps reprocessing harmless.
  • Page cap is 500 rows/tick. A workspace processing > 500 routine runs/minute at sample_rate = 1.0 will accumulate backlog; the watermark only advances past successfully-handled rows, so the backlog drains across subsequent ticks rather than being lost.

​
Validation gates and credential leak guards

Each step’s validation block runs after the step output materializes. The schema is a JSON Schema draft 2020-12 subset (most of type, required, properties, items, pattern, format, enum, etc.) plus three Crewship extensions:
  • must_not_contain: ["API_KEY=", "Bearer "] — output must include none of these substrings
  • must_contain: ["##"] — output must include all of these
  • min_length / max_length — convenience for non-JSON outputs
The must_not_contain gate is the credential-leak tripwire: if an agent is about to leak a real API key in its output, the gate fails the step before downstream consumers see it. Pair with on_fail: "abort" (set at the step level, alongside validation) for hard stop, or escalate_tier if you want the higher model to retry without the leak.

​
Observability

Every routine run emits a sequence of journal entries:
  • pipeline.run.started — once at run begin
  • pipeline.step.started — per step
  • pipeline.step.container_ready — per agent step: how long the step spent acquiring its crew container (duration_ms). This isolates the container-provision cost from the LLM/tool time in the step’s total duration — so you can read the effect of container prewarming directly (a warm hit is near-zero, a cold provision is seconds) instead of guessing under LLM-latency noise. A transiently-retried step emits one record per attempt, distinguished by an attempt field in the payload. Visible in routine logs <run_id> --full (kept off the compact activity rail).
  • pipeline.step.completed / pipeline.step.failed / pipeline.step.validation_failed — per step terminal
  • pipeline.step.skipped — the step’s if condition was false, so it never ran (payload carries the condition). A dedicated entry type rather than a completed row with a marker, so a skipped branch is distinguishable from a real completion at the storage layer.
  • pipeline.step.retrying — a transient failure the retry policy is about to swallow; non-terminal, one per attempt (attempt/max), followed by the step’s eventual completed/failed. (Rows written before this type existed arrive as pipeline.step.completed/failed carrying a kind=skipped/kind=retry payload marker; readers honour that marker as a fallback.)
  • pipeline.run.completed / pipeline.run.failed — once at run end
  • summary.generated — an LLM-generated outcome verdict (“did the run accomplish the goal?”), fired once after the run terminates. Shown as an outcome pill next to the status pill in the Runs sub-tab. Gated by the run_verdict_summaries feature flag and skipped for agentless/dry-run routines — see the Crew Journal guide for the full contract.
Plus WebSocket broadcast on the workspace channel for live UI updates (PipelineRunNode in the orchestration graph + Runs sub-tab waterfall both subscribe).

​
The step’s chat is a durable transcript

Every agent_run step gets its own chat (titled Pipeline <id> · step <id>, one per attempt), and that chat keeps both halves of the turn: the rendered prompt the step sent, and the agent’s answer with its reasoning and tool activity in the order they streamed. Read it with crewship chat <chat-id>, or open the chat in the dashboard — the transcript is the same one crewship chat stream shows live, so a browser left open on a running step still has the text after a reload. A step that failed after the agent said something keeps what it said, which is usually the most useful thing on the screen. These chats raise no “agent replied” inbox item, by design: nobody asked the step a question, and a routine on a fifteen-minute cron would otherwise bell you ninety-six times a day per step. Use a notify step when a run genuinely needs to reach someone.

​
Live visibility — what is a routine doing right now?

While at least one routine run is in flight, the dashboard surfaces it from anywhere in the app:
  • Header chip — a pulsing “N routines running” pill appears in the toolbar next to the Online / Crews pills (hidden when nothing is active). If any run is parked on a human approval it turns amber and appends ”· M awaiting approval”. Clicking opens a popover with the six newest active runs — routine name, short run id, elapsed time, cost so far, current step — each with Open trace ↗ (deep-link to /activity?run=<id>), Cancel (same manage-tier RBAC as the Runs tab), and a Review → shortcut into the routine for parked runs. With more than six active runs a “View all N running →” footer jumps to the Activity rail pre-filtered to the active bucket (/activity?status=active).
  • /routines sidebar — a routine with an active run gets a pulsing blue dot and a sub-line showing the current step and elapsed time (▶ ask-casey · 0:12); a parked run shows the amber ⏸ awaiting approval variant.
  • /routines list table — the status cell swaps the historical “last run” pill for a live Running (or amber Awaiting approval) pill with current step · elapsed · cost, and live routines bubble to the top of the table regardless of the chosen column sort.
All three surfaces share one workspace-scoped subscription (GET /api/v1/workspaces/{ws}/pipeline-runs?status=active + the pipeline.run.*/pipeline.step.started events, 3s poll while anything is active). status=active bundles running, queued, paused and waiting — the status a run parks in while a HITL waitpoint awaits a decision.

​
Run warnings

before_all/after_all/on_failure lifecycle hooks run best-effort: a failing after_all or on_failure hook (a teardown step like credential-release or cost-meter-close) never flips the run’s terminal status — a before_all failure is different and fails the run outright, since nothing downstream ran. A failed after_all/on_failure hook is instead recorded as a structured warning on the run so it doesn’t silently vanish into server logs while the run reports completed. Fetch it via GET /api/v1/workspaces/{ws}/pipeline-runs/{runId} — the response’s warnings array (always present, empty when there are none) has one entry per failed hook:
crewship routine logs <run_id> (the slug-free state lookup) prints a Warnings: section when the run has any, alongside the existing Error: line for the run’s own terminal status.

​
Per-step cost + duration

Both the UI Runs sub-tab waterfall and crewship routine logs <run_id> --slug X surface the cost_usd and duration_ms fields the executor stamps on every pipeline.step.completed event. Same data, two presentation surfaces:
  • UI: right-aligned columns next to each step row, with a footer total summing the run. An em-dash (—) renders for events that don’t carry cost (pipeline.step.started, .failed, live-only echoes) — easier to scan than $0.0000 next to real values.
  • CLI logs: extra DURATION and COST columns in the timeline output. Same em-dash rule for non-positive values.
Worker tier escalation is visible too — a failed validation gate that retries on a higher tier emits a second step.started+step.completed pair with its own cost on the retry, so the column-summed footer reflects the full spend (including retries), not just the first attempt.

​
Monthly budget meter

Per-step cost is the trace-level view; monthly budget (#1422 item 3) is the operator-level one — a spend cap you set for a routine, compared against actual pipeline_runs.cost_usd for the current calendar month:
This is not the same knob as DSL max_cost_usd (above) — that’s a per-RUN hard gate authored into the definition and enforced mid-run by the executor (a run aborts if it would breach it). The monthly budget is external operator config, set out-of-band from the DSL (never touched by routine save, never bumps the version history), and is a view + soft cap indicator, not an enforcement gate — a routine over its monthly budget keeps running; nothing currently blocks the next invocation. A routine can have neither, either, or both knobs. routine budget summary rolls up every routine that has a budget set OR spend this month (a routine with neither is excluded — noise reduction for workspaces with dozens of zero-cost deterministic routines):
The routine detail page shows the same meter (spent-vs-cap bar) when a budget is set, and the workspace roll-up appears on the routines list page. API: GET/PATCH /api/v1/workspaces/{ws}/pipelines/{slug}/budget, GET /api/v1/workspaces/{ws}/pipelines/budget-summary — see the CLI reference for the full flag/response shape.

​
Run tags & metadata

Tag and annotate a run at invoke time to make it filterable later:
Tags are workspace-scoped labels (max 10/run, lowercased) surfaced on the run detail and usable as a filter (crewship routine runs <slug> --tag prod); replays inherit the source run’s tags. Metadata is a free-form JSON object stored on the run and returned by GET /api/v1/workspaces/{ws}/pipeline-runs/{runId} — set at invoke time today (mid-run mutation and {{ run.metadata.X }} templating are a follow-up).

​
Replay & error fingerprinting

Failed runs are bucketed by a stable error fingerprint (failing step + normalized message), so recurring failures group together instead of scrolling past one-by-one in run history:
A replay is stamped is_replay=true + replay_of=<run_id>; gate a step on {{ env.is_replay }} to skip a side effect (e.g. a notification) on replay. Full reference: Run observability: tags, metadata, replay, errors. Three terminal-side observability surfaces, ordered by when you’d reach for each: For variance characterisation across many runs (is this routine production-ready at this tier?), use crewship routine bench. For matrix-level cross-tier consistency (which scenarios diverge between Haiku and Opus?), use crewship eval scenarios and crewship eval baseline diff for CI regression gates.

​
RBAC

​
Common patterns

​
”Run this routine every day at 9 AM"

​
"Trigger this routine from a GitHub Actions hook"

​
"Validate routine DSL in CI before committing”

Exit code 1 on any invalid file. Beyond the schema, validate catches at author time what used to fail at save or on the first run: a concurrency_key that can render empty, unsatisfiable output gates (min_length > max_length, same token in must_contain and must_not_contain), and dead egress_targets entries (*, *.*, empty host — targets are literal/suffix matched, not globbed). Agent-slug typos fail here too if you supply the roster: --agents triage,writer (fully offline) or --author-crew growth (one GET /api/v1/agents call); the two flags are mutually exclusive. Details in the CLI reference.

​
”Discover what an agent is about to do, without running it”

Returns a would_execute report with which agent, which tier, the rendered prompt, and an estimated cost. Zero side effects.

​
”Watch a metric on a tight cron, spend tokens only on a spike”

Author an agentless probe that prints true/false, then attach it as a wake gate:
The probe runs every 5 minutes for free; the LLM routine fires only on the ticks where the probe’s output is truthy. routine schedules list shows how often the gate woke vs. checked.

​
“Get a daily ops digest of runs, cost, and failures”

The seeded workspace-digest routine (query → transform → notify, agentless, zero cost) covers this out of the box — wire a cadence with:
Creates the routine (if a prior nuke removed it — --crew names who’ll own it) and a daily 08:00 UTC schedule, idempotently. See the Digest CLI reference for --cron/--when and how delivery to Slack/email works (it’s the same notify step + notification-preference matrix every other routine uses).

​
”Cancel an in-flight run”

Signals the run goroutine; it stops at the next safe point and emits pipeline.run.failed with reason “cancelled”.

​
”Serialise runs per tenant / customer / repo”

Use concurrency_key with a template referencing the tenant-identifying input:
Two requests for the same account_id queue rather than fan out; requests for different account_ids run in parallel. Pair with the Idempotency-Key header for webhook handlers (see the Concurrency + idempotency recipe). The platform fails fast if the rendered key is empty (missing input + no literal prefix in the template) — see Troubleshooting for why and how to fix.

​
Troubleshooting

Before going through this list: run crewship routine doctor <slug> first. Most “blind alley” failures (missing crew provisioning, agent slug typo, missing credential, contradictory validation gate, cost cap too tight) surface as a ✗ check on doctor before they ever cost an LLM call.
The save endpoint requires the test-gate cleared, and the server no longer trusts a body “it passed” claim — the old last_test_run_at / last_test_run_passed fields are ignored (they were forgeable). The gate is cleared by an HMAC save_token: POST the draft to the public, JWT-authed POST /api/v1/workspaces/{ws}/pipelines/test_run first — it parses + Validates + checks the integration/resource preconditions + dry-runs the DSL (no agent invoked) and, on pass, mints a token bound to (workspace, definition_hash, user). Forward that token in the /save body as "save_token". crewship routine save performs this two-step (test_run → save) for you, so you normally never see this 422 from the CLI — it surfaces when a hand-rolled /save call omits the token.Alternative: crewship apply --skip-test-gate (CLI — the flag lives on apply, not routine save) / "skip_test_gate": true (API) if your role is OWNER/ADMIN and you trust the DSL — bypasses the gate explicitly.
Check that ?include_steps=1 is in the runs URL the UI fetches. The list endpoint defaults to run-level only to keep payload bounded; the detail panel passes the flag explicitly. After a refresh, the waterfall populates from journal entries plus live WebSocket events.
The scheduler ticks every 30s; with single-binary deployment, restarting the server resets the in-memory tick cursor. Pending schedules whose next_run_at has passed will fire on the next tick after restart. If you’ve edited the cron expression, next_run_at recomputes from now() — so a 0 9 * * * schedule edited at 10:30 won’t fire until 9 AM tomorrow.
HMAC mismatch: check the X-Crewship-Signature: sha256=<hex> header, computed as HMAC-SHA256(signing_secret, request_body) over the raw bytes the sender sent (not a re-serialized form). The server uses hmac.Equal for comparison so timing-safe.
Two-tier escalation walks the fallback chain. With on_fail: escalate_tier, a failed Haiku run retries on Sonnet (5-10× more expensive), then Opus (20-50× more expensive) before giving up. Tighten validation gates (loosen must_contain, raise min_length), or set max_cost_usd on the routine to abort the run between steps when a cost ceiling is hit.
Since resume-from-step landed, a restart between the wait step starting and a decision arriving re-attaches the run to its pending waitpoint at boot — approving via crewship routine waitpoints approve <token> resumes it. If the run shows interrupted instead, its persisted state was insufficient to resume (the reason is in the run’s error_message); the orphaned waitpoint can still be listed and rejected to clear the inbox. The boot log lines pipeline boot recovery done (resume-from-step) resumed=N interrupted=M and pipeline waitpoint store wired (...) stranded_pending=N show what recovery did.
The DSL declared a non-empty concurrency_key template but the rendered value is an empty string — typically because a referenced input was omitted at trigger time. Full error message:
Why fail-fast: a routine that declares concurrency_key: "{{ inputs.account_id }}" is asking the platform to serialise runs per tenant. If account_id is missing, treating the empty key as “no gate” would silently allow unlimited parallelism for a routine the author explicitly asked us to serialise — a denial-of-self by misconfiguration. The executor refuses to start the run.Fixes, in order of preference:
  1. Supply the input. The caller (curl / crewship routine run --inputs '{...}' / scheduler / webhook) needs to pass the referenced input. For webhooks this means the inputs_template in the webhook config must produce it from the incoming payload.
  2. Set a default on the InputSpec. If the input is genuinely optional but you still want a tenant-style gate, add "default": "global" (or any non-empty sentinel) to the InputSpec. The platform merges defaults before rendering the key.
  3. Use a literal prefix. A template like "vendor-alert-{{ inputs.vendor_id }}" always renders non-empty (the literal vendor-alert- survives even when vendor_id is missing); the key still gates, just less granularly.
  4. Drop the gate. If you genuinely don’t want concurrency control, omit concurrency_key entirely (do NOT set it to "" — that’s the unset sentinel).
Catch it before you ship. crewship routine validate (and save) now reject a concurrency_key that can render fully empty — one built entirely from {{ inputs.X }} refs where none is required: true/defaulted. Applying fix #2 (anchor a referenced input) or fix #3 (a literal prefix) satisfies the check, so the empty-render failure is caught at author time instead of on the first bad run.

​
CLI reference

For batch evaluation across the eval-* fleet (matrix sweep, head-to-head tier compare, baseline regression diff), see the eval CLI. Cookbook recipe 6 walks the eval-driven promotion workflow end-to-end. The pipeline alias is preserved for back-compat on the legacy subcommands (pipeline list, pipeline run, pipeline get, etc.). The post-rename additions — schedules, webhooks, waitpoints, validate, watch, logs, records, bench, doctor — are only registered under routine, so scripts that need them must switch.

​
Backend reference

  • Migrations v78 (pipelines + execution_tiers_json), v79 (versions + waitpoints), v80 (schedules), v81 (run idempotency), v82 (webhooks), v115 (schedule wake gates), v153 (schedule circuit breaker), v154 (signal waits)
  • Source: internal/pipeline/ (~10 700 LOC, 36 files)
  • API: internal/api/pipelines.go, pipeline_runs.go, pipeline_schedules.go, pipeline_webhooks.go
  • Sidecar: internal/sidecar/pipelines.go (port 9119)
  • Frontend: app/(dashboard)/routines/, components/features/routines/, hooks use-pipelines*

​
Production notes

Two caveats worth internalizing before you lean on routines for anything time- or cost-sensitive — both are permanent architectural properties of the current single-binary deployment, not bugs waiting on a fix:
Single-instance only. The scheduler, run registry, and online eval watermark all assume one process. Running multiple replicas would double-fire cron schedules and webhooks (no leader election yet) — see Cron schedules.
Crash recovery is at-least-once, step-granular, not exact. A hard kill loses at most the in-flight step, which re-executes (and re-bills) on resume. call_pipeline has no nested checkpointing — a kill mid-nested-run re-executes the entire nested pipeline. max_cost_usd under-counts after a crash, since whatever the killed step had already spent isn’t in the restored total. See Durability and restart recovery for the full resume matrix.

​
Limitations (current MVP)

These are known gaps in the current MVP — none block production use, but they shape what you can rely on today.
  • Resume is at-least-once, step-granular — restart recovery re-enters runs from the last persisted step boundary (see Durability and restart recovery). The step that was in flight at the kill re-executes from scratch; there is no sub-step checkpointing, nested call_pipeline runs re-execute in full, and DAG runs recover at wave granularity. Runs whose definition changed since the run started (content-hash mismatch) fall back to interrupted. max_cost_usd under-counts true spend after a crash — see Production notes.
  • Single-instance scheduler — running multiple replicas would double-fire schedules.
  • credentials_required scope field is documentary — the type is enforced at run time (a live run is refused with 422 + missing_credentials when the vault lacks an ACTIVE credential of that type — see Required credentials), but the optional per-entry scope is not yet used for finer-grained matching.
  • No cross-adapter tier swap yet — same-provider model swap (Haiku→Opus) works; Claude→Gemini swap requires a shorthand→constant mapping not yet wired.
  • NL→cron covers a fixed phrase set — routine schedules create --when (see Cron schedules) handles the common onboarding phrasings (every weekday at 9, every day at 9am, every monday at 14:00, every hour, every N minutes/hours) via a small hand-rolled parser, not a general NLP model; anything outside that set still needs a raw --cron expression.