Paymaster
Paymaster owns cost accounting and budget enforcement. It is the read/write side of thecost_ledger and budget_limits tables (migration 52, extended by migration 62) and it ships as middleware on the LLM call stack. Every priced LLM round-trip produces exactly one cost_ledger row plus a llm.call journal entry; metered rows additionally emit cost.incurred, and either side can emit budget.warning / budget.exceeded when limits are hit.
Paymaster separates two billing models that cannot share a single $ aggregate:
- Metered — pay-per-token API-key calls. Real $ math, real budgets.
- Flat-rate — subscription credentials (Anthropic Max, ChatGPT Plus, etc.). The $200/mo is paid up front; the marginal cost of one more token is $0 from our perspective.
billing_mode column and the UI keeps the two surfaces disjoint: KPIs and budgets show metered spend; subscriptions get a separate panel with call/token counts but no $ figure.
Data model
cost_ledger + budget_limits schema (migration 52, extended by v62)
cost_ledger + budget_limits schema (migration 52, extended by v62)
Budget scopes
Budgets apply at four levels, narrowing from workspace down to agent. All applicable budgets are checked before every call; the most restrictive breach wins:
Windows:
hour, day, week, month, and mission (window-less; sums the whole mission regardless of duration). Hour/day/week/month are calendar-aligned in UTC — dashboards reset on the hour, at 00:00 UTC, Monday 00:00 UTC, and the 1st respectively.
Enforcement modes
- soft —
Enforcenever returns an error. Emitsbudget.warningwhen over 100% so the UI still paints it red. Use for cost-curious teams that want visibility without the platform pulling the plug. - hard —
BudgetExceededErrorat 100%. No warn band. Emitsbudget.exceeded. - tiered — emits
budget.warningat 80% (no block) andbudget.exceededat 100% (block). The default for new budgets; gives operators a window to react.
Billing modes
Everycost_ledger row is tagged metered or flat_rate. The orchestrator picks the mode from the credential type before exec and signals it to the sidecar via CREWSHIP_BILLING_MODE (and CREWSHIP_SUBSCRIPTION_PLAN for flat-rate). The sidecar copies the values onto every POST /api/v1/internal/cost/record it sends to the server.
Record enforces the flat-rate invariants regardless of caller input — the LiteLLM-style “NULL beats fake-$0” pattern, adapted to SQLite’s NOT NULL. A flat-rate row is still a complete audit trail (which credential, which agent, which mission, when, how many tokens), it just refuses to invent a dollar figure.
The cost.incurred journal entry is suppressed for flat-rate rows because no money was incurred. llm.call still emits, with the summary explicitly tagged (flat-rate · <plan>) so operators glancing at the timeline aren’t misled.
Cost confidence
Three values, written to every row:precise— provider returned a usage block we trust verbatim (e.g. Anthropic non-streaming).estimate— we computed it frompricing.gorates against parsed token counts.unknown— flat-rate row, or we couldn’t parse the response body (for example an opaque proxy response or a stream with no terminal usage event).
Quota enforcement
The sidecar parses two header conventions on every upstream response:anthropic-ratelimit-{requests,input-tokens,output-tokens}-{limit,remaining,reset}x-ratelimit-{remaining-tokens,remaining-requests,reset-tokens}(OpenAI / xAI / DeepSeek)
remaining / limit ratio is surfaced as a single fraction. Once parsed, the sidecar calls EnforceQuota (after Record):
Quota signals are per-call, ephemeral — they are journaled but not persisted to a side table.
EnforceQuota is called after Record so the row that triggered the signal is still present in cost_ledger for forensics.
This is the only enforcement path that fires for flat-rate calls: subscription users hit provider rate limits before they hit a $ ceiling. The error type is intentionally identical so *BudgetExceededError handlers do not need to know whether the cause was $ or quota.
Prompt-cache token flow
Cache token counts are provider-reported and surface as discounted ledger lines. The wire path:1
Provider returns usage
Anthropic emits
cache_read_input_tokens + cache_creation_input_tokens in the usage block (both non-streaming and message_start SSE events); OpenAI emits prompt_tokens_details.cached_tokens (no separate creation counter — caching is opaque on their side).2
internal/llm/{anthropic,openai}.go parses
The fields into
Response.CachedInputToks + Response.CacheCreationToks.3
The OpenAI codec subtracts
OpenAI’s
prompt_tokens includes the cached read, so the codec stores prompt_tokens - cached_tokens (clamped at zero) in Response.InputToks. Anthropic’s input_tokens is already disjoint from cache_read_input_tokens and is taken verbatim.4
internal/llm/middleware.go plumbs
Them through
providerCaller / streamCaller into paymaster.CallResponse.CachedInputTokens + .CacheCreationTokens.5
paymaster.Middleware records
Them into the matching
cost_ledger columns alongside the rate-card snapshot. Cache reads bill at rate_cached_in_per_m; cache creates at rate_cache_write_per_m.6
telemetry.LLMMiddleware stamps
The same counts as
gen_ai.usage.cached_input_tokens + gen_ai.usage.cache_creation_tokens on the llm.call span — see Tracing.claude-haiku-4-5 cache read bills 1.00/M base input) lands automatically because the rate-card column is its own field; no per-row multiplier needed.
input_tokens on a ledger row means FRESH input on every provider — exclusive of cached_input_tokens and cache_creation_tokens. That convention is a provider-side subtraction, not an Estimate one: Estimate bills the three channels additively, so subtracting inside it would under-bill every Anthropic call by cached × input_rate and double-subtract the sidecar path, which has always corrected its own numbers (internal/sidecar/usage.go).This changed for OpenAI-family rows. Before it, prompt_tokens went to the ledger verbatim and the cached read was billed twice — once at the full input rate, once at the cached rate — 2.67× over on an 80%-cached 1500-token prompt. Historic rows are not migrated, so an OpenAI rollup spanning the change is not comparable across it, and the cache-hit ratio’s denominator shrank (a ratio above 1 is now possible for OpenAI, as it always was for Anthropic).cost_ledger.cached_input_tokens column has been present since v62 but was effectively dead until this wiring landed.
Cost controls
Beyond the ledger and budgets, a few knobs cut token spend directly without touching output quality.Cache-stable system prompt
Anthropic bills a cache read at ~10% of a fresh input token, but only when the request’s prefix is byte-identical to a recent one. The orchestrator keeps the--system-prompt (preamble → persona → skills → memory files) stable within a day and pushes everything that changes turn-to-turn — conversation history, episodic recall, the [MEMORY NUDGE] and [COST AWARENESS] blocks — into a [SESSION CONTEXT] wrapper on the user message instead. Dynamic content in the system prefix used to force a full-price re-read on every message; moving it out lets the large stable prefix hit the cache. Watch cost_ledger.cache_creation_tokens (should fall) vs cached_input_tokens (should rise) across a multi-turn session to confirm the win.
Auxiliary & sub-agent model routing
Auxiliary work (memory consolidation, Keeper/behavior evaluators, memory-health, negative learning) ships defaulted toanthropic/claude-haiku-4-5, but an unconfigured slot follows the credential you actually have — see Evaluator defaults follow the credential you actually have. Override any slot per deployment — point it at a cheaper or local model without a redeploy:
CREWSHIP_AUX_CURATOR_MODEL=llama3.1 CREWSHIP_AUX_CURATOR_PROVIDER=ollama.
Delegated worker sub-agents (a lead handing a bounded sub-task to a crew member) rarely need the top model tier. Set CREWSHIP_SUBAGENT_MODEL to route them to a cheaper model; the lead planner keeps its own configured model. Unset = each agent uses its own model (no downgrade), so this is a pure opt-in.
Turn caps & loop guard
Two independent guards stop a confused agent from burning budget:--max-turns— the adapter-side loop cap. Interactive runs default to50; scheduled / routine runs default to20(orchestrator.RoutineMaxTurns) because an unattended job with no human watching is where a stuck loop runs up the bill unnoticed. Override it per-run from the CLI:crewship run <agent> --max-turns 15 "..."(also oncrewship ask).0(the default) leaves the built-in cap in place.- Loop guard — the orchestrator aborts a run when the agent repeats the identical tool call (same name + same input) five times in a row, emitting an
exec.commandjournal entry withreason: loop_detectedso cost triage can tell a runaway loop apart from a real crash. Adapter-agnostic — it observes the normalized tool-call stream, so it covers every CLI.
Rate-card snapshotting
Every metered row writes therate_*_per_m columns from RateCard(provider, model) at the moment of write. When the rate card later changes, historical rollups stay consistent: the old row continues to imply its old rate. There are now two ways it can change — an edit to pricing.go (a provider repriced, a new model added), and a refresh of the embedded models.dev snapshot (go generate ./internal/modelcatalog/...), which can reprice any model that resolves at step 3 of the lookup below without anyone touching pricing.go. New rollup queries can either trust cost_usd directly or recompute from rate_*_per_m * tokens to verify nothing has drifted.
Zero is the honest value for ollama/local (genuinely free) and for unknown-provider lookups (we couldn’t price). The columns are stored as nullable REAL (the v62 migration adds them without a NOT NULL / DEFAULT constraint), but the writer always supplies an explicit zero in those cases, so historical rows never observe NULL in practice — readers can treat the column as effectively populated.
Pricing — current rates
The canonical rate card lives ininternal/paymaster/pricing.go. Adding a model is a one-line change. Prices are USD per 1,000,000 tokens and were verified against provider docs on 2026-04-30.
It is no longer the only rate source: an embedded models.dev snapshot (internal/modelcatalog, 8 providers) sits underneath it and prices models the table has never heard of. The table stays authoritative — see The five-step lookup — and crewship model price --provider <p> --model <m> reports which source answered for any pair.
Rate card — per-provider prices (USD / 1M tokens)
Rate card — per-provider prices (USD / 1M tokens)
Anthropic Opus 4.7 was repriced to $5/$25 from the old $15/$75 in early 2026. The Crewship rate card was updated on 2026-04-30; ledger rows written before that date carry the old rate as a snapshot in
rate_*_per_m and will continue to read out at the historical price. New rows use the corrected rate.The five-step lookup
priceTable is no longer the only rate source. lookupPrice resolves (provider, model) in this order and stops at the first hit:
The ceiling (step 4) picks the most-expensive known tier rather than the median, so budgets warn correctly even on premium reasoning models, at the cost of a mild over-estimate for cheap unknown ones. Better to over-estimate than to silently bill $0.
The snapshot sits third on purpose: after the exact match, because
priceTable carries verified corrections a bulk import must never overwrite (Opus 4.7 was billed 3× over until someone checked); and after the wildcard, so a future refresh that pulled an ollama or local provider into the trim could not start billing local calls.
Models the snapshot carries no cost for — including 23 all-zero cost blocks, which are upstream gaps rather than free models — are skipped, never written as a $0 row, and fall through to the ceiling.
The call path
paymaster.Middleware(next, j, db) wraps an LLMCaller. Order of operations:
1
Enforce
Check resolves every applicable budget; a hard/tiered breach short-circuits with *BudgetExceededError. The underlying call is never made, no ledger row is written, but budget.exceeded lands in the journal.2
Call
Delegate to the inner caller.
3
Record
Fill
cost_usd via Estimate(provider, model, tokens) if the provider didn’t price the call inline, then INSERT the row and emit llm.call + cost.incurred.telemetry -> paymaster -> lookout -> raw).
Read endpoints
GET /api/v1/paymaster/spend/by-crew?range=7d— one row per crew (metered only).GET /api/v1/paymaster/spend/by-agent/{crewId}?range=24h— per-agent rollup inside a crew (metered only).GET /api/v1/paymaster/spend/by-mission/{missionId}— single-mission total (metered only).GET /api/v1/paymaster/top-spenders?limit=10&range=7d— highest-cost scopes (metered only).GET /api/v1/paymaster/subscriptions?range=30d— flat-rate credentials grouped by(plan, provider)with call count, token count, last-used timestamp. No $ figure — see “Billing modes” above for why.
range=1h|24h|7d|30d or explicit since=<RFC3339>&until=<RFC3339>. Default is 7 days.
The four spend/* endpoints filter WHERE billing_mode = 'metered' so flat-rate rows do not silently inflate metered totals. The Subscriptions panel is the only place flat-rate rows surface to the operator.
Internal write endpoint
The sidecar writes ledger rows over the IPC socket, never directly. The handler isPOST /api/v1/internal/cost/record (auth: X-Internal-Token). Authoritative scope (workspace / crew / agent IDs) comes from the sidecar’s IPCConfig set at exec time, so an agent that captured the token still cannot forge cross-tenant attribution. Mirrors the /internal/journal/emit security model — see Internal IPC API.
Workspace isolation: the by-agent and by-mission handlers reject cross-tenant IDs with 404 (same shape as “not found”) so existence isn’t leaked. See crewBelongsToWorkspace / missionBelongsToWorkspace in internal/api/paymaster_handler.go.
CLI
crewship paymaster.
Creating a budget
Example workspace-wide hard cap of $250/day:Sidecar coverage of agent CLI calls
Paymaster wrapsllm.Provider.Complete() for everything called from the Go side (summaries, Keeper, consolidation, quartermaster judge). For agent CLI calls (claude, gemini, cursor-agent, droid) the sidecar is the metering point: the proxy intercepts the outbound HTTPS request, parses the response body for usage tokens, harvests rate-limit headers, and async-POSTs a Call to POST /api/v1/internal/cost/record. The handler validates auth and calls Record + EnforceQuota.
What the sidecar can and cannot see:
For pinned-cert tunnels the sidecar emits a “credential was used” attribution row (
billing_mode=flat_rate, no tokens, cost_confidence=unknown) so the Subscriptions panel still shows usage. We deliberately do not MITM with a custom CA — CLIs with pinned certs would reject it, and the honest answer is that the body is private.
Body observation uses io.TeeReader with a 10 MB cap so streaming UX is preserved while a pathological upstream cannot OOM the sidecar. Both JSON and SSE responses continue to pass through after the cap; only bytes beyond it are omitted from usage parsing.
Look for the llm.call journal entry: if it exists, the call was metered. If you see exec.command with claude and no llm.call, the OAuth tunnel was used and the row will land as flat-rate.
Gotchas
- Soft budgets still emit entries. A soft budget at 120% still writes
budget.warningto the journal — the UI uses this to paint red. Don’t filter out “soft” when looking for “who was over”. - Stream path is unmetered.
wrappedProvider.Stream()bypasses the full middleware (token counts arrive in the terminalmessage_deltaevent, which the syncCallResponseshape can’t carry). Streaming callers pay through orchestrator-level accounting that predates this package. - Pricing is estimated, and a “price” may be a ceiling. A new model from a provider the snapshot covers (anthropic, openai, google, deepseek, mistral, xai, openrouter, amazon-bedrock) is priced automatically at step 3. Anything else lands on the per-provider ceiling at step 4 — a real number, deliberately high, and not the invoice. The estimate does not fall to $0 unless the provider id itself is unknown, which is the case worth chasing: a registry row or a
Provider.Name()with no rate row and no catalog entry bills every call at $0. Runcrewship model priceon the pair before assuming;pricing.gois where a correction goes, but it is no longer the first place to look.
Related
- LLM middleware — layer order and composition.
- Multi-provider LLM configuration — the registry, the codecs, and the embedded catalog that supplies step 3 of the lookup.
crewship paymaster— CLI.- Paymaster API.
- Crew Journal —
llm.call,cost.incurred,budget.*entry types.