crewship routine save, validate with crewship eval scenarios.
If you’re new to routines, start with the routines guide for the conceptual model. This page assumes you know what agent_run, complexity, and validation are.
How to read these recipes
Every recipe has the same anatomy:- Goal — what the routine does and when it’s worth using.
- DSL — the full JSON definition. No omissions, no
.... - Gates explained — why each
must_contain/ schema /outcomes.criteriais there. The gates are the design — copying the prompt without the gates doesn’t reproduce the routine. - Failure modes & fixes — what goes wrong on weak tiers and the typical resolution.
The gates are the design. Copying a recipe’s prompt without its
validation / outcomes block does not reproduce the routine — the gates are what make the output reliable enough to depend on.complexity: fast (Haiku, cheap) and complexity: smart (Opus, robust).
Recipe 1 — Strict JSON extraction
Goal: turn freeform text into a typed JSON object that the next step cantransform over without parsing prose. The canonical “make this prompt deterministic enough to be a function” pattern.
Gates explained
must_containon{,},"item","qty","unit_price","currency"— anchors on JSON structure + every required key name. Catches the most common failure: model emits prose like “Sure, here’s the JSON: …” before the actual object.must_not_containon```,Here is— blocks code-fence wrappers and conversational lead-ins.max_length: 400— caps verbosity drift. A weak tier sometimes ignores “no prose” and adds a paragraph of explanation; the cap trips it.validation.schema(JSON Schema draft-2020-12) — full schema walk is enforced when set, compiled and cached per definition byinternal/pipeline/executor_validate.go. Use it when you want type-checking; the substring anchors above are the cheap-but-effective first layer that short-circuits the expensive compile + walk on obvious failures.
Failure modes & fixes
Recipe 2 — Cross-family rubric grading
Goal: a fast worker drafts the output, a smart grader scores it on a strict rubric, the loop iterates if the worker misses the rubric. Mitigates self-preference bias by using a different grader family.Why this pattern
A bare gate (must_contain: ["- "]) catches the bullet marker but not “did the worker actually summarise the topic, or just emit three placeholder bullets?” The rubric’s covers_topic + no_invented_facts criteria are what prove the output is grounded. The grader is a separate agent_slug (eva, Sonnet) so its self-preference bias doesn’t pile onto the worker’s (daniel, Haiku).
Gotchas
max_iterations: 3— the loop is bounded so a stubborn worker can’t burn unbounded tokens. After 3 attempts,on_fail: abortlets the run fail honestly rather than ship a bad summary.max_cost_usd: 1.50— outcomes loops can iterate up to 3× the worker cost + 1 grader call per iteration. Budget accordingly.- Don’t put more than ~10 criteria in one rubric — the grader’s verdict gets noisy. Split into two graded steps if you need finer-grained rubric.
Recipe 3 — Tier escalation on cost guardrail
Goal: pin a routine to the cheapest tier that satisfies the gate, but escalate automatically if the cheap tier fails. Production routines should have this — it’s how you catch a model regression without a 4am page.Escalation flow
- Worker runs at
fasttier (Haiku). Output failsmust_containbecause it added “I think the sentiment is…” on_fail: escalate_tier→ executor walksexecution_tier.fallback, retries onmoderate(Sonnet).- Sonnet’s output passes the gate. Run completes with
cost_usd≈ Haiku-cost + Sonnet-cost. - Journal records both attempts so you can see which tier actually satisfied the gate.
When NOT to use auto-escalation
- Critical-output routines where wrong-but-confident is worse than failed. Rubric-graded scenarios with
outcomes.on_fail: abortare better — failing loud is preferable to silently spending more. - Cost-budget-pinned routines where you’d rather see
FAILED: cost cap exceededthan auto-bump to a $0.20 tier you didn’t budget for.
Recipe 4 — DAG with deterministic transform plumbing
Goal: combine non-LLM steps (http, transform, code) with LLM steps to keep cost down where determinism is achievable.
Why the transform step matters
Withoutproject_title, the LLM step would see the entire JSON response (often KBs of data) and either truncate context or summarise the wrong field. A deterministic transform step (jq projection) reduces cost AND eliminates a class of hallucination — the LLM only sees what we explicitly extract.
Egress allowlist
egress_targets: ["api.example.com"] is enforced at runtime. A typo’d URL host fails at the http step rather than going to a different server.
Recipe 5 — Idempotent routine with concurrency key
Goal: a webhook-triggered routine that should never double-execute on retransmission. The pattern follows the standard webhook idempotency model — pair an idempotency key with a concurrency limit so retries are safe and a burst of duplicate events doesn’t fan out to N parallel executions.Idempotency vs concurrency — what’s the difference?
concurrency_keygates parallel runs: if two requests with the same key arrive at the same time, the second one waits (or 429s) until the first finishes.Idempotency-KeyHTTP header dedupes across time: a second request with the same key (within the TTL) returns the originalrun_idwithstatus=DEDUPEDinstead of executing again.
concurrency_key validation
The platform fails fast when a non-empty concurrency_key template renders to an empty string. Example: a routine declares concurrency_key: "{{ inputs.order_id }}" but the caller triggers without supplying order_id — the rendered key is "", which would otherwise be treated as “no gate” and silently allow unlimited parallelism. The executor instead returns:
"global"); if you want no gate, omit the field entirely.
Triggering safely
Idempotency-Key, the second request returns status=DEDUPED and the original run id. No double-charge, no double-fulfilment.
Recipe 6 — Eval-driven promotion to production
Goal: take a hand-written routine and decide, with data, whether to ship it onfast or smart tier. This is the workflow PRD §18 calls “operator promotion path.”
What to do at each verdict
Recipe 7 — Cross-tier compare (head-to-head)
Goal: investigate one specific scenario when the matrix fromeval scenarios shows divergence.
DIVERGE-B-PASS) tells you Haiku can’t satisfy the gate but Opus can. Now you have a real data point for the “ship at fast?” decision: probably no, unless you can rewrite the gate.
Recipe 8 — Routine that calls another routine
Goal: compose a complex workflow from smaller, individually-tested routines. Each sub-routine has its own gates, costs, baseline.Why compose
- Each sub-routine has its own benched baseline. A regression in
summarize-eventsshows up as a regression in any composed routine that calls it — you don’t have to re-bench the parent. - Author-crew context is preserved per call. If
summarize-eventslives in thequalitycrew, calling it from aengineeringroutine still runs withquality’s persona. - Cycle detection: A → B → A is rejected at save time (even a pair authored in the wrong order — the being-saved draft is fed back into the check), and a runtime guard rejects any cycle that still slips through before it churns to the depth ceiling. Maximum nested depth is 10.
- The parent’s budget bounds the child: a
call_pipelinechild inherits the parent’s remainingmax_cost_usdand stops when it’s exhausted, rather than counting from zero against only its own cap.
Recipe 9 — Fan out over a list with foreach + CEL gating
Goal: process every element of an array concurrently — one HTTP fetch + one summary per URL — then keep only the non-trivial results with a CEL if:. The per-item outputs collect into a single array you can hand to a final step.
How it runs
eachrenders{{ inputs.urls }}to a JSON array and runs the two-step body once per URL, up to 6 in flight at a time (parallelism). Each URL is bound as{{ inputs.url }}inside the body.- The body’s
summarizestep gates on a CEL expression —inputs.url != ""— a real comparison, not a template. Before CEL support,"if": "inputs.url != \"\""rendered to the literal string and always read as truthy; now it evaluates. each’s output is the JSON array of every item’s last body-step output, in input order:["summary 1","summary 2",…]. Thedigeststep’s own CELif:—steps.each != "[]"— skips the newsletter when the fan-out produced nothing.- Cost is summed across all items and attributed to
each, so the routine-levelmax_cost_usd: 1.50bounds the entire fan-out — a runaway 500-URL input stops at the budget instead of spending unbounded.
Gotchas
itemsmust render to a JSON array. Anobjector scalar is a hard error; an empty array is a valid no-op ([]).- No
wait,call_pipeline, or nestedforeachinside the body — keep the fan-out bounded and self-contained. Reach forcall_pipelineat the top level if an item needs to invoke a whole other routine. - Fan-out is fail-fast: the first item that fails cancels the rest and fails the step. Wrap a flaky body step in
retry:if transient failures shouldn’t sink the batch.
Recipe 10 — Incremental sync with a cross-run watermark
Goal: a scheduled routine that processes only what’s new since the last run — the classic watermark pattern. Each run reads the last cursor from{{ routine.state.* }}, does its work, and writes the new cursor back with state_write for the next run.
How the watermark flows
- First run:
{{ routine.state.last_id }}is empty, sosince_id=fetches from the beginning. Theprocessstep writeslast_id= the highest id it handled. - Every run after: reads the
last_idthe previous run wrote, fetches only newer records, and advances the watermark. A CELif:(steps.fetch != "[]") skips the write-back entirely on an empty batch, so the cursor never regresses. - Per-schedule isolation: if you attach a second schedule (say a manual backfill schedule), it keeps its own
last_id— the two never clobber each other. Manualcrewship routine runcalls share one default bucket. - Restart-safe: the cursor is in the database, not memory. A process restart between runs loses nothing.
Watermark + wake gate
Because the state bucket is shared with the schedule’s wake gate, an agentless probe can read the same watermark to decide whether there’s anything new before the (paid) main routine fires — attach the probe when you create the schedule:Recipe 11 — Backtest a candidate version before rollout
Goal: you editedsupport-triage (new prompt, tightened gate, whatever) and saved it as v9. Before it becomes the version live traffic gets, prove it behaves at least as well as v8 did — on real recent inputs, not a hand-picked eval suite.
REGRESSION_DETECTED — wire it into CI the same way as eval baseline diff:
Why this is read-only
backtest never creates a pipeline version and never changes head_version — it replays each corpus run’s original captured inputs pinned to the immutable version you named with --against (the same pinning mechanism cron schedules and webhooks already use for target_pipeline_version). Live traffic keeps resolving to whatever head_version already was, before, during, and after the backtest. If it reports CLEAN, roll forward with routine rollback or your normal promotion path — backtest only tells you whether it’s safe, it doesn’t flip anything.
What to do at each verdict
How this differs from Recipe 6
Recipe 6 (eval-driven promotion) validates against an authored eval suite — scenarios you wrote, run fresh against each tier.backtest validates against actual production inputs already captured for this routine — no authoring required, and it catches regressions an eval suite’s fixed scenario set wouldn’t (a real customer message the gate didn’t anticipate). Use both: eval suite for the tier decision, backtest as the pre-rollout gate for every subsequent version bump.
Recipe 12 — Human approval on the write
Goal: let an agent decide something genuinely ambiguous, but never let it touch the outside world unreviewed. This is the shape to reach for whenever a routine both reasons and acts: triaging an inbound issue and labelling it, drafting a customer reply and sending it, classifying an alert and paging someone. The pattern is four beats — fetch deterministically → reason → gate → act — and the order is the design.Gates explained
fetch_issue + issue_facts come before the agent, not inside it. If the agent gathers its own facts, it will eventually invent one — a version number, a stack trace, a file path. Deterministic steps mean same issue in, same facts out, every run. The agent gets a clean context instead of a research task, which is also why complexity: fast is enough here.
classify is the only step that reasons, and its json_schema is what makes its output safe to interpolate downstream. Without the schema, publish would POST whatever prose came back.
review is the whole point. A wait step of kind: approval parks the run — the goroutine releases its concurrency slot, the run row goes to waiting, and an inbox card appears. Nothing after this step runs until a human decides. The park survives a restart: the waitpoint is a row, not a timer.
publish sits after review in the DAG. This is the load-bearing bit. Every step capable of changing something outside Crewship belongs downstream of the gate. Put publish in needs: ["classify"] instead and the gate becomes decoration — the run would publish and then ask.
on_fail: abort on the gate is not optional
A wait step whose timeout_seconds elapses falls through to its on_fail. With abort, an approval nobody answered within 24 hours fails the run and nothing is published. That is the behaviour you want: an unanswered question is not a yes.
Failure modes & fixes
When the gate stops earning its keep
The honest failure mode of this pattern is not that it is unsafe — it is that it gets ignored. Approve the same routine twenty times with the same verdict and the twenty-first approval carries no judgement; you are rubber-stamping, and a rubber stamp is worse than no gate because it looks like review. Once you have approved the same gate on the same routine body three times, the inbox card starts offering a standing grant: the gate stops asking, and says so in the audit trail rather than quietly.- Per gate, not per routine. Trusting
reviewdoes not trust adelete_everythinggate in the same routine. - Pinned to the routine’s definition hash. Edit the routine — any step, any prompt — and the hash moves, no grant matches, and the gate asks again. You cannot be talked into trusting a gate and then have the step under it rewritten.
trust listmarks grants whose definition has moved on asstale. - Still a real approval. A grant fires by writing an ordinary approved waitpoint attributed to the operator who granted it, with the grant id in its decision payload. The run history shows an approval that happened and who stands behind it — there is no invisible path through the gate.
- Ignored by strict crews. A crew at
autonomy_level: stricthas opted out of every shortcut around the operator, and standing grants are one. The crew that governs is the routine’s author crew, since a routine executes in its author’s context.
Grants are for gates that have become repetitive, not for gates that have become inconvenient. If a gate is slow because the reviewer disagrees with the agent half the time, the answer is a better prompt or a tighter schema — not a grant.
What’s intentionally NOT in the cookbook
These exist in the routines documentation but aren’t recipes here because they don’t change the patterns above:- Schedules — see routine schedules CLI. Add a cron and you have a periodic version of any recipe above.
- Webhooks — see routine webhooks CLI. HMAC-signed event triggers for any recipe above.