Routines

​

What is a Routine?

• A JSON DSL definition — inputs, outputs, ordered or DAG-structured steps, validation gates, declared egress, credential requirements
• Authorship metadata — which crew, which agent (or user), via which path
• Triggers — cron schedules and webhook tokens that fire it autonomously
• Run history — every invocation as an immutable journal trace with step-level events
• Versions — every save creates a new version; rollback is one API call

​

Three authoring paths

​

DSL anatomy

{
  "dsl_version": "1.0",
  "name": "summarize-text",
  "description": "Summarize input text in 3 bullets.",
  "inputs": [
    { "name": "text", "type": "string", "required": true }
  ],
  "outputs": [
    { "name": "summary", "type": "string" }
  ],
  "steps": [
    {
      "id": "summarize",
      "type": "agent_run",
      "agent_slug": "tomas",
      "complexity": "fast",
      "prompt": "Summarize: {{ inputs.text }}"
    }
  ]
}

​

Top-level fields

​

InputSpec

{
  "name": "since",
  "type": "string",
  "required": false,
  "default": "yesterday",
  "description": "ISO date or 'yesterday'/'today'",
  "min": 0,
  "max": 100
}

​

Template substitution

• inputs.X — declared input value
• steps.Y.output — full text output of an earlier step
• steps.Y.output.path — JSON path into a step’s output (when output parses as JSON)
• env.AUTHOR_CREW_NAME / etc. — read-only allowlist of execution context

​

Required integrations

{
  "name": "triage-incident",
  "integrations_required": ["slack", "github"],
  "steps": [ ... ]
}

• Declared is always allowed. Saving a routine that names an integration the crew lacks is fine — declaring is a contract, not a connection. Only the run enforces. (Save-time validation only checks the list is well-formed: non-empty slugs, lowercased/trimmed, within a sane count cap.)
• Enforced at run time. Before a run starts, Crewship resolves the integrations the routine’s author crew has connected and compares them to integrations_required. If any are missing the run is blocked with an RFC 7807 Problem Details response, HTTP 422, carrying a machine-readable missing_integrations: string[] member and a human detail like routine requires integration "slack" not connected for crew "Marketing". The UI uses missing_integrations to render a Connect action. The run never starts — no tokens spent.
• No-op fast path. An empty / absent integrations_required does zero resolution work — no overhead for routines that don’t use it.
• Fail-open. If integration-availability resolution itself errors, the run is allowed (a warning is logged). A bug in resolution must never wedge every run of every routine — a forgotten integration is a soft, recoverable failure; a hard block on all runs would be a self-inflicted outage.
• run is gated; dry_run is not. A live run executes against the author crew’s agents, so it’s gated (fail fast rather than land an unrunnable routine). The internal save gate’s draft validation applies the same integration check. dry_run is a preview that invokes nothing, so it’s left ungated — it shows what the routine would need, even integrations the crew hasn’t connected yet.

​

Capability manifest

​

The resources block

{
  "name": "deploy-service",
  "resources": {
    "datastores": [
      { "type": "postgres", "name": "main", "note": "writes table runs" },
      { "type": "redis",    "name": "cache" }
    ],
    "tools": [
      { "type": "ansible",  "name": "deploy.yml" },
      { "type": "kubectl" }
    ]
  },
  "steps": [ ... ]
}

• datastores[].type — any string, but use the canonical vocabulary redis | postgres | mysql | mongodb | other so it matches what the crew’s container catalog reports (the precondition gate compares type case-insensitively; a non-canonical type simply won’t match a crew resource).
• tools[].type — any string; canonical examples ansible | terraform | kubectl | bash | python | other. Same matching rule as datastores.
• name / note are free-form and optional.

​

Required resources

• Enforced at run time. Before a run starts, Crewship resolves the author crew’s container resources and compares them to the declared resources.datastores + resources.tools. If the routine requires a datastore or tool the crew doesn’t have, the run is blocked with an RFC 7807 Problem Details response, HTTP 422, carrying a machine-readable missing_resources: [{ kind, type, name }] member (kind is "datastore" or "tool") and a human detail like routine needs datastore postgres, tool ansible, not available to crew "Ops". The run never starts — no tokens spent — until the resource is provisioned on the crew.
• Matching. A required datastore is satisfied when the crew has a datastore of the same engine type (case-insensitive; the service name is advisory and not matched). A required tool is satisfied when the crew has an installed tool whose name matches the required type (the tool’s name, e.g. deploy.yml, is the concrete artifact and isn’t matched against the container).
• No-op fast path. A routine that declares no resources block does zero resolution work. Only the declared resources are gated — code-step runtimes folded into the manifest’s tools (e.g. cel) are internal executor runtimes, not crew CLIs, and are never gated.
• Fail-open. If the routine has no author crew, or resource resolution itself errors, the run is allowed (a warning is logged) — same reasoning as the integration gate: a resolver bug must never wedge every run.
• run is gated; dry_run is not. (The internal save gate’s draft validation applies the same resource check.)

​

Agents already know what the container has

​

Step types

​

agent_run

{
  "id": "summarize",
  "type": "agent_run",
  "agent_slug": "tomas",
  "complexity": "fast",
  "prompt": "Summarize: {{ inputs.text }}",
  "model_override": null,
  "timeout_seconds": 60,
  "validation": {
    "schema": { "type": "object", "required": ["title"] },
    "must_not_contain": ["API_KEY=", "Bearer "],
    "min_length": 30
  },
  "on_fail": "escalate_tier"
}

​

call_pipeline

{
  "id": "review_summary",
  "type": "call_pipeline",
  "pipeline_slug": "human-approval-step",
  "nested_inputs": {
    "content": "{{ steps.summarize.output }}",
    "approver_role": "marketing_lead"
  }
}

​

http

{
  "id": "fetch",
  "type": "http",
  "http": {
    "method": "GET",
    "url": "{{ inputs.url }}",
    "max_response_bytes": 200000,
    "success_codes": [200],
    "credential_ref": { "type": "API_KEY", "inject_as": "bearer" }
  },
  "timeout_seconds": 30
}

1. Routine layer — egress_targets. When the routine declares egress_targets, the request host must be one of the declared hosts or a subdomain of one (api.x.com matches target x.com; evilx.com does not). A routine that declares no egress_targets is unrestricted at this layer — backward-compatible with every routine that predates the field. The SSRF guard (private/link-local IPs, DNS-rebind-safe dialing) applies regardless.
2. Crew layer — network policy. The authoring crew’s network policy (network_mode + allowed domains — the same dial that governs the crew’s agent containers) also applies to direct http steps. Crews on the default free mode are unaffected; a restricted crew’s routines can only reach the crew’s allowed domains (exact host match, same as the container proxy).

​

wait

{
  "id": "human_approval",
  "type": "wait",
  "wait": {
    "kind": "approval",
    "approval_prompt": "Approve summary?",
    "timeout_sec": 86400
  }
}

​

code

​

runtime: expr (wired, token-zero)

- id: probe
  type: code
  code:
    runtime: expr
    code: "{{ inputs.spend_usd }} > {{ inputs.threshold_usd }}"

• Operators: > >= < <= == !=.
• The body is Render()-ed first, so {{ inputs.x }} / {{ steps.y.output }} placeholders substitute before evaluation.
• Anything that isn’t a single comparison (multiple operators, function calls, arbitrary code) fails closed with a clear error — expr is deliberately not a scripting language.

​

runtime: cel (wired, token-zero, general logic)

- id: spike
  type: code
  code:
    runtime: cel
    # Real logic in one deterministic, token-zero step — no {{ }} needed,
    # inputs are exposed as a typed `inputs` map:
    code: 'inputs.spend_usd > inputs.threshold_usd && inputs.region in ["eu", "us"]'

​

runtime: bash | python | go (rejected at author time)

- id: process
  type: code
  code:
    runtime: python
    code: "import json; print(json.dumps({'sum': sum(input_nums)}))"

​

transform

{
  "id": "extract_emails",
  "type": "transform",
  "transform": {
    "input": "{{ steps.fetch.output }}",
    "expression": ".items[] | .email"
  }
}

​

Conditional if

​

DAG with needs[]

{ "id": "fetch_a", "type": "http", "http": {...} },
{ "id": "fetch_b", "type": "http", "http": {...} },
{ "id": "merge",
  "type": "agent_run",
  "needs": ["fetch_a", "fetch_b"],
  "prompt": "Merge: {{ steps.fetch_a.output }} + {{ steps.fetch_b.output }}"
}

​

Lifecycle hooks (before_all / after_all / on_failure)

hooks:
  before_all:   # setup — its FAILURE aborts the run before any step executes
    id: claim
    type: http
    http: { method: POST, url: "https://ops.example.com/claim" }
  after_all:    # runs after a COMPLETED run — best-effort
    id: release
    type: http
    http: { method: POST, url: "https://ops.example.com/release" }
  on_failure:   # runs after a FAILED run — best-effort cleanup
    id: alert
    type: http
    http: { method: POST, url: "https://ops.example.com/alert" }

​

Per-step overrides (no version bump)

crewship routine step-override set my-routine summarize \
  --prompt "Summarize in 3 bullets, lead with the risk." --model smart
crewship routine step-override list my-routine
crewship routine step-override clear my-routine summarize

​

Agentless routines

{
  "dsl_version": "1.0",
  "name": "cost-spike-probe",
  "agentless": true,
  "egress_targets": ["billing.example.com"],
  "steps": [
    { "id": "fetch", "type": "http",
      "http": { "method": "GET", "url": "https://billing.example.com/today" } },
    { "id": "amount", "type": "transform",
      "transform": { "input": "{{ steps.fetch.output }}", "expression": ".spend_usd" } },
    { "id": "judge", "type": "code",
      "code": { "runtime": "expr", "code": "{{ steps.amount.output }} > 100" } }
  ]
}

• Save time — validation rejects agent_run (direct LLM spend), call_pipeline (the target resolves by slug at runtime, so a nested routine could gain an agent step later and silently break the guarantee), and eval.online with sample_rate > 0 (online grading runs a grader agent against the routine’s runs).
• Run time — the executor independently refuses to dispatch an LLM-capable step inside an agentless run, so even a definition written before the validator existed fails closed.

​

Two-tier execution

{
  "trivial":  { "primary": {"adapter":"claude","model":"claude-haiku-4-5-20251001"} },
  "fast":     { "primary": {"adapter":"claude","model":"claude-haiku-4-5-20251001"},
                "fallback":[{"adapter":"claude","model":"claude-sonnet-4-6"}] },
  "moderate": { "primary": {"adapter":"claude","model":"claude-sonnet-4-6"} },
  "smart":    { "primary": {"adapter":"claude","model":"claude-opus-4-7"} }
}

​

Save validation gate

​

Governance — agent proposes, human approves the risky ones

​

Maker-checker on save

• it declares an integrations_required the author crew can’t currently satisfy (the same resolver the run gate uses);
• it has any http/egress step (or routine-level egress_targets);
• it has any code-runtime step;
• it declares credentials_required.

• Safe → active. Goes live immediately, exactly as before.
• Risky → proposed. The routine is persisted but not runnable, and a blocking inbox item is raised for MANAGER+ (the same Inbox surface as proposed skills). Approve it to go live, or reject it.

​

Approve / reject (MANAGER+)

crewship routine approve <slug>   # status → active, resolves the inbox item
crewship routine reject  <slug>   # removes the routine, resolves the inbox item

• POST /api/v1/workspaces/{ws}/pipelines/{slug}/approve — MANAGER+. Flips to active.
• POST /api/v1/workspaces/{ws}/pipelines/{slug}/reject — MANAGER+. Soft-deletes the proposed routine.

​

Disable / enable (OWNER/ADMIN airbag)

crewship routine disable <slug>   # status → disabled; cancels in-flight runs
crewship routine enable  <slug>   # status → active

• POST /api/v1/workspaces/{ws}/pipelines/{slug}/disable — OWNER/ADMIN. Flips to disabled and cancels any in-flight runs of that routine immediately.
• POST /api/v1/workspaces/{ws}/pipelines/{slug}/enable — OWNER/ADMIN. Returns it to active.

crewship routine list --status proposed

​

Triggers

​

Cron schedules

crewship routine schedules create --slug summarize-text \
    --name "daily-summary" --cron "0 9 * * *" --timezone "Europe/Prague" \
    --inputs '{"text":"…default"}'

​

Wake gates

# LLM only runs when the probe prints something truthy:
crewship routine schedules create --slug cost-report \
    --cron "*/15 * * * *" \
    --wake-slug cost-spike-probe --wake-inputs '{"threshold":"100"}'

# Drop the gate later (fire on every tick again):
crewship routine schedules update <schedule_id> --no-wake

• The probe must be agentless: true, live in the same workspace, and can’t be the schedule’s own routine — all validated when the schedule is saved.
• Probe errors fail open: a broken or deleted probe wakes the main routine instead of going silently blind, and records last_wake_status: ERROR so you can see the probe needs fixing.
• A skipped tick advances next_run_at but leaves last_run_* untouched — run telemetry stays strictly about main runs. Wake telemetry lives in wake_check_count / wake_fire_count / last_wake_at / last_wake_status, and routine schedules list shows it as <probe> woke/checked in the WAKE column.
• Probe executions are regular runs with triggered_via: wake_check, so they’re auditable in run history and filterable out of dashboards.

​

Webhooks

crewship routine webhooks create --slug pr-review-structured \
    --hmac-secret "$(openssl rand -hex 32)" --rate-limit 30

HTTP/1.1 202 Accepted
{"run_id": "run_c...", "status": "PENDING", "deduped": false}

crewship routine logs <run_id>          # persisted run state (status, step outputs, error)
crewship routine logs <run_id> --full   # full step-by-step journal timeline
# or: GET /api/v1/workspaces/{wsId}/pipeline-runs/{runId}

​

Manual

​

Deferred dispatch: delay, ttl, debounce, priority

# fire 60s from now, expire if not dispatched within 5 min, high priority
crewship routine run my-routine --delay 60 --ttl 300 --priority 9

# coalesce a burst: repeated triggers sharing a key collapse into one run
crewship routine run my-routine --debounce-key vendor-42 --debounce-window 30 --debounce-max 300

crewship routine pending list            # not-yet-fired deferred triggers
crewship routine pending cancel <id>     # cancel before it fires

​

Dry-run preview

crewship routine dry-run email-fetch --inputs '{"since":"yesterday"}'

• Step ID + type
• Resolved tier_adapter:tier_model (e.g. claude:claude-haiku-4-5)
• Estimated cost in USD (order-of-magnitude only — the executor uses a flat token-density heuristic, not real pricing)
• would_call_agent / would_call_pipeline target

​

Versioning + rollback

crewship routine versions email-fetch
crewship routine rollback email-fetch --to 3

​

Bundle export / import

# Export from workspace A
crewship routine export email-fetch --include-history > email-fetch.json

# Import into workspace B
cat email-fetch.json | crewship routine import

​

HITL waitpoints

# CLI
crewship routine waitpoints list
crewship routine waitpoints approve <token> --comment "LGTM"
crewship routine waitpoints reject  <token> --comment "needs revision"

# UI: /routines → routine → Wait sub-tab → click Approve / Reject
# API: POST /api/v1/workspaces/{ws}/pipelines/waitpoints/{token}/approve

​

Durability and restart recovery

• Completed steps are restored, not re-executed. Their outputs feed downstream {{ steps.X.output }} templates exactly as if the process had never died.
• The in-flight step re-executes from scratch — at-least-once semantics. For an agent_run step this means the agent call is re-issued (and re-billed); http/code steps with external side effects may fire twice. Design steps to be idempotent where that matters.
• Runs parked on a wait approval step re-attach to the original waitpoint token. No duplicate approval card is created; the pending inbox item stays answerable across the restart, and approving it resumes the run.
• DAG runs (needs:) resume at wave granularity — the parallel scheduler flushes state when each wave completes, so a kill mid-wave re-executes that wave’s unfinished steps.
• call_pipeline boundaries are NOT persisted. A kill while a nested pipeline is executing re-runs the entire nested pipeline on resume — the parent’s call_pipeline step is the unit of recovery, and the nested run’s own per-step progress is not checkpointed. Keep nested pipelines short or idempotent if a mid-flight kill matters to you.
• The accumulated cost is restored too, so max_cost_usd keeps counting across the restart instead of resetting. Caveat: cost is flushed at step boundaries, so whatever the killed in-flight step had already spent before the kill is not in the restored total — the cap under-counts the true spend by up to one step’s worth (and the re-executed step is billed again on top).
• A resumed run that finds its concurrency slot occupied waits and retries with capped exponential backoff (2s doubling up to 60s) instead of failing — losing the slot race to a freshly-fired scheduled run is a timing collision, not a reason to abandon hours of restored work. If the server shuts down while a run is still waiting for its slot, the row stays in-flight and the next boot resumes it again.
• A waitpoint that timed out while the process was down resumes, observes the expired token, and fails with wait step "X" (approval) timed out — distinct from an operator clicking deny (… denied).

​

Online eval sampler

name: nightly-summary
eval:
  online:
    sample_rate: 0.05              # grade 5% of runs; 0 disables, 1.0 grades every run
    grader_agent_slug: qa-grader   # references an agent in the AUTHOR crew with a rubric prompt

1. Scan pipeline_runs WHERE status = 'completed' AND completed_at > watermark.
2. For each candidate, resolve the routine DSL. If eval.online is absent or sample_rate <= 0, skip.
3. Draw from crypto/rand. If the sample lands above sample_rate, skip.
4. Otherwise, INSERT into eval_runs with kind = 'online', status = 'queued'. The existing grader worker picks it up and writes the result back.

• Schema-layer idempotency — eval_runs has a partial UNIQUE INDEX (pipeline_run_id) WHERE kind='online'. A duplicate sampler instance or a crash-recovery watermark replay can attempt the same enqueue twice; the second collapses to a no-op rather than queueing twice and double-billing the grader.
• (completed_at, id) tuple cursor — parallel fan-out steps that complete at the same nanosecond all get graded; a timestamp-only cursor would orphan siblings.
• Stuck-on-error watermark — a transient per-row error (resolver outage, entropy outage, enqueue conflict) freezes the watermark at the row before the failure so the next tick retries. Deterministic skips (no eval config, sample roll missed) advance normally.
• Trace correlation — the eval row carries routine_slug + pipeline_run_id so an operator clicking a low-scoring grade in the eval UI lands on the actual trace via pipeline_run_id -> trace.

• Watermark is in-memory only. On process restart it resets to now - 1h; outages longer than that leave a gap in grading coverage. The UNIQUE index keeps reprocessing harmless.
• Page cap is 500 rows/tick. A workspace processing > 500 routine runs/minute at sample_rate = 1.0 will accumulate backlog; the watermark only advances past successfully-handled rows, so the backlog drains across subsequent ticks rather than being lost.

​

Validation gates and credential leak guards

• must_not_contain: ["API_KEY=", "Bearer "] — output must include none of these substrings
• must_contain: ["##"] — output must include all of these
• min_length / max_length — convenience for non-JSON outputs

​

Observability

• pipeline.run.started — once at run begin
• pipeline.step.started — per step
• pipeline.step.completed / pipeline.step.failed / pipeline.step.validation_failed — per step terminal
• pipeline.run.completed / pipeline.run.failed — once at run end

​

Live visibility — what is a routine doing right now?

• Header chip — a pulsing “N routines running” pill appears in the toolbar next to the Online / Crews pills (hidden when nothing is active). If any run is parked on a human approval it turns amber and appends ”· M awaiting approval”. Clicking opens a popover with the six newest active runs — routine name, short run id, elapsed time, cost so far, current step — each with Open trace ↗ (deep-link to /activity?run=<id>), Cancel (same manage-tier RBAC as the Runs tab), and a Review → shortcut into the routine for parked runs. With more than six active runs a “View all N running →” footer jumps to the Activity rail pre-filtered to the active bucket (/activity?status=active).
• /routines sidebar — a routine with an active run gets a pulsing blue dot and a sub-line showing the current step and elapsed time (▶ ask-casey · 0:12); a parked run shows the amber ⏸ awaiting approval variant.
• /routines list table — the status cell swaps the historical “last run” pill for a live Running (or amber Awaiting approval) pill with current step · elapsed · cost, and live routines bubble to the top of the table regardless of the chosen column sort.

​

Run warnings

{ "stage": "hook after_all", "message": "credential release step timed out", "at": "2026-07-02T12:01:05Z" }

​

Per-step cost + duration

• UI: right-aligned columns next to each step row, with a footer total summing the run. An em-dash (—) renders for events that don’t carry cost (pipeline.step.started, .failed, live-only echoes) — easier to scan than $0.0000 next to real values.
• CLI logs: extra DURATION and COST columns in the timeline output. Same em-dash rule for non-positive values.

​

Run tags & metadata

crewship routine run cost-spike-probe \
  --tag prod --tag billing \
  --metadata '{"source":"manual","ticket":"OPS-42"}'

​

Replay & error fingerprinting

crewship routine replay <run_id>                   # re-invoke with the run's captured inputs
crewship routine errors                             # list fingerprint groups
crewship routine bulk-replay --fingerprint <fp>      # replay every run in a fingerprint group after a fix

​

RBAC

​

Common patterns

​

”Run this routine every day at 9 AM"

crewship routine schedules create --slug daily-status-digest \
    --cron "0 9 * * *" --timezone "Europe/Prague"

​

"Trigger this routine from a GitHub Actions hook"

crewship routine webhooks create --slug pr-review-structured \
    --hmac-secret "$(openssl rand -hex 32)" --rate-limit 30
# Use the printed token + secret as Settings → Webhooks in the GH repo.

​

"Validate routine DSL in CI before committing”

- run: crewship routine validate routines/*.json

​

”Discover what an agent is about to do, without running it”

crewship routine dry-run summarize-text --inputs '{"text":"..."}'

​

”Watch a metric on a tight cron, spend tokens only on a spike”

crewship routine schedules create --slug incident-summary \
    --cron "*/5 * * * *" --wake-slug error-rate-probe

​

”Cancel an in-flight run”

crewship routine cancel <run_id>

​

”Serialise runs per tenant / customer / repo”

{
  "concurrency_key": "{{ inputs.account_id }}",
  "max_concurrent": 1,
  "inputs": [
    { "name": "account_id", "type": "string", "required": true }
  ]
}

​

Troubleshooting

pipeline: concurrency_key rendered to empty value (referenced input missing or empty): template "{{ inputs.<NAME> }}"

​

CLI reference

​

Backend reference

• Migrations v78 (pipelines + execution_tiers_json), v79 (versions + waitpoints), v80 (schedules), v81 (run idempotency), v82 (webhooks), v115 (schedule wake gates)
• Source: internal/pipeline/ (~10 700 LOC, 36 files)
• API: internal/api/pipelines.go, pipeline_runs.go, pipeline_schedules.go, pipeline_webhooks.go
• Sidecar: internal/sidecar/pipelines.go (port 9119)
• Frontend: app/(dashboard)/routines/, components/features/routines/, hooks use-pipelines*

​

Production notes

​

Limitations (current MVP)

• Resume is at-least-once, step-granular — restart recovery re-enters runs from the last persisted step boundary (see Durability and restart recovery). The step that was in flight at the kill re-executes from scratch; there is no sub-step checkpointing, nested call_pipeline runs re-execute in full, and DAG runs recover at wave granularity. Runs whose definition changed since the run started (content-hash mismatch) fall back to interrupted. max_cost_usd under-counts true spend after a crash — see Production notes.
• Single-instance scheduler — running multiple replicas would double-fire schedules.
• credentials_required is declared-only — unlike integrations_required and resources, it is not yet enforced at run time (see Required integrations).
• No cross-adapter tier swap yet — same-provider model swap (Haiku→Opus) works; Claude→Gemini swap requires a shorthand→constant mapping not yet wired.
• No NL→cron converter — ops still hand-type 0 9 * * *. Foundation PRD has the design; not in MVP.