Skip to main content

Paymaster

Paymaster owns cost accounting and budget enforcement. It is the read/write side of the cost_ledger and budget_limits tables (migration 52, extended by migration 62) and it ships as middleware on the LLM call stack. Every priced LLM round-trip produces exactly one cost_ledger row plus a llm.call journal entry; metered rows additionally emit cost.incurred, and either side can emit budget.warning / budget.exceeded when limits are hit. Paymaster separates two billing models that cannot share a single $ aggregate:
  • Metered — pay-per-token API-key calls. Real $ math, real budgets.
Proxied agent calls recorded $0 until August 2026. parseLLMUsage matched on a lowercase provider name while the sidecar proxy passed an uppercase one, so no proxied response body was ever parsed and every such call wrote a zero-token, $0 row. Any rollup, KPI or budget decision taken on proxied traffic before that fix was taken against zero.The fix means those crews now post real token counts. Budgets that never fired on them can fire on the first day — the spend did not change, only the recording of it. Ceilings set while the rows read $0 should be re-checked.
  • Flat-rate — subscription credentials (Anthropic Max, ChatGPT Plus, etc.). The $200/mo is paid up front; the marginal cost of one more token is $0 from our perspective.
Mixing them in the same KPI silently misleads operators, so the ledger row carries a billing_mode column and the UI keeps the two surfaces disjoint: KPIs and budgets show metered spend; subscriptions get a separate panel with call/token counts but no $ figure.

Data model

Budget scopes

Budgets apply at four levels, narrowing from workspace down to agent. All applicable budgets are checked before every call; the most restrictive breach wins: Windows: hour, day, week, month, and mission (window-less; sums the whole mission regardless of duration). Hour/day/week/month are calendar-aligned in UTC — dashboards reset on the hour, at 00:00 UTC, Monday 00:00 UTC, and the 1st respectively.

Enforcement modes

  • softEnforce never returns an error. Emits budget.warning when over 100% so the UI still paints it red. Use for cost-curious teams that want visibility without the platform pulling the plug.
  • hardBudgetExceededError at 100%. No warn band. Emits budget.exceeded.
  • tiered — emits budget.warning at 80% (no block) and budget.exceeded at 100% (block). The default for new budgets; gives operators a window to react.
A tiered budget ties one emit per tick to the state it’s in, so a once-warned call stays in the journal even if the subsequent call pushes it over the line.

Billing modes

Every cost_ledger row is tagged metered or flat_rate. The orchestrator picks the mode from the credential type before exec and signals it to the sidecar via CREWSHIP_BILLING_MODE (and CREWSHIP_SUBSCRIPTION_PLAN for flat-rate). The sidecar copies the values onto every POST /api/v1/internal/cost/record it sends to the server. Record enforces the flat-rate invariants regardless of caller input — the LiteLLM-style “NULL beats fake-$0” pattern, adapted to SQLite’s NOT NULL. A flat-rate row is still a complete audit trail (which credential, which agent, which mission, when, how many tokens), it just refuses to invent a dollar figure. The cost.incurred journal entry is suppressed for flat-rate rows because no money was incurred. llm.call still emits, with the summary explicitly tagged (flat-rate · <plan>) so operators glancing at the timeline aren’t misled.

Cost confidence

Three values, written to every row:
  • precise — provider returned a usage block we trust verbatim (e.g. Anthropic non-streaming).
  • estimate — we computed it from pricing.go rates against parsed token counts.
  • unknown — flat-rate row, or we couldn’t parse the response body (for example an opaque proxy response or a stream with no terminal usage event).
The dashboard surfaces these as badges next to $ figures. Aggregate KPIs deliberately do not mix confidences — when a rollup spans multiple bands, the lowest band wins so the operator sees the floor of certainty, not the average.

Quota enforcement

The sidecar parses two header conventions on every upstream response:
  • anthropic-ratelimit-{requests,input-tokens,output-tokens}-{limit,remaining,reset}
  • x-ratelimit-{remaining-tokens,remaining-requests,reset-tokens} (OpenAI / xAI / DeepSeek)
The most restrictive window’s remaining / limit ratio is surfaced as a single fraction. Once parsed, the sidecar calls EnforceQuota (after Record):
Quota signals are per-call, ephemeral — they are journaled but not persisted to a side table. EnforceQuota is called after Record so the row that triggered the signal is still present in cost_ledger for forensics. This is the only enforcement path that fires for flat-rate calls: subscription users hit provider rate limits before they hit a $ ceiling. The error type is intentionally identical so *BudgetExceededError handlers do not need to know whether the cause was $ or quota.

Prompt-cache token flow

Cache token counts are provider-reported and surface as discounted ledger lines. The wire path:
1

Provider returns usage

Anthropic emits cache_read_input_tokens + cache_creation_input_tokens in the usage block (both non-streaming and message_start SSE events); OpenAI emits prompt_tokens_details.cached_tokens (no separate creation counter — caching is opaque on their side).
2

internal/llm/{anthropic,openai}.go parses

The fields into Response.CachedInputToks + Response.CacheCreationToks.
3

The OpenAI codec subtracts

OpenAI’s prompt_tokens includes the cached read, so the codec stores prompt_tokens - cached_tokens (clamped at zero) in Response.InputToks. Anthropic’s input_tokens is already disjoint from cache_read_input_tokens and is taken verbatim.
4

internal/llm/middleware.go plumbs

Them through providerCaller / streamCaller into paymaster.CallResponse.CachedInputTokens + .CacheCreationTokens.
5

paymaster.Middleware records

Them into the matching cost_ledger columns alongside the rate-card snapshot. Cache reads bill at rate_cached_in_per_m; cache creates at rate_cache_write_per_m.
6

telemetry.LLMMiddleware stamps

The same counts as gen_ai.usage.cached_input_tokens + gen_ai.usage.cache_creation_tokens on the llm.call span — see Tracing.
The 10% discount for Anthropic cache reads (a claude-haiku-4-5 cache read bills 0.10/Mvs0.10/M vs 1.00/M base input) lands automatically because the rate-card column is its own field; no per-row multiplier needed.
input_tokens on a ledger row means FRESH input on every provider — exclusive of cached_input_tokens and cache_creation_tokens. That convention is a provider-side subtraction, not an Estimate one: Estimate bills the three channels additively, so subtracting inside it would under-bill every Anthropic call by cached × input_rate and double-subtract the sidecar path, which has always corrected its own numbers (internal/sidecar/usage.go).This changed for OpenAI-family rows. Before it, prompt_tokens went to the ledger verbatim and the cached read was billed twice — once at the full input rate, once at the cached rate — 2.67× over on an 80%-cached 1500-token prompt. Historic rows are not migrated, so an OpenAI rollup spanning the change is not comparable across it, and the cache-hit ratio’s denominator shrank (a ratio above 1 is now possible for OpenAI, as it always was for Anthropic).
A regression earlier in this PR’s history shipped the parser without consuming the cache fields — every workspace recorded zero cached tokens and the discount never triggered. The cost_ledger.cached_input_tokens column has been present since v62 but was effectively dead until this wiring landed.

Cost controls

Beyond the ledger and budgets, a few knobs cut token spend directly without touching output quality.

Cache-stable system prompt

Anthropic bills a cache read at ~10% of a fresh input token, but only when the request’s prefix is byte-identical to a recent one. The orchestrator keeps the --system-prompt (preamble → persona → skills → memory files) stable within a day and pushes everything that changes turn-to-turn — conversation history, episodic recall, the [MEMORY NUDGE] and [COST AWARENESS] blocks — into a [SESSION CONTEXT] wrapper on the user message instead. Dynamic content in the system prefix used to force a full-price re-read on every message; moving it out lets the large stable prefix hit the cache. Watch cost_ledger.cache_creation_tokens (should fall) vs cached_input_tokens (should rise) across a multi-turn session to confirm the win.

Auxiliary & sub-agent model routing

Auxiliary work (memory consolidation, Keeper/behavior evaluators, memory-health, negative learning) ships defaulted to anthropic/claude-haiku-4-5, but an unconfigured slot follows the credential you actually have — see Evaluator defaults follow the credential you actually have. Override any slot per deployment — point it at a cheaper or local model without a redeploy:
Example: CREWSHIP_AUX_CURATOR_MODEL=llama3.1 CREWSHIP_AUX_CURATOR_PROVIDER=ollama. Delegated worker sub-agents (a lead handing a bounded sub-task to a crew member) rarely need the top model tier. Set CREWSHIP_SUBAGENT_MODEL to route them to a cheaper model; the lead planner keeps its own configured model. Unset = each agent uses its own model (no downgrade), so this is a pure opt-in.

Turn caps & loop guard

Two independent guards stop a confused agent from burning budget:
  • --max-turns — the adapter-side loop cap. Interactive runs default to 50; scheduled / routine runs default to 20 (orchestrator.RoutineMaxTurns) because an unattended job with no human watching is where a stuck loop runs up the bill unnoticed. Override it per-run from the CLI: crewship run <agent> --max-turns 15 "..." (also on crewship ask). 0 (the default) leaves the built-in cap in place.
  • Loop guard — the orchestrator aborts a run when the agent repeats the identical tool call (same name + same input) five times in a row, emitting an exec.command journal entry with reason: loop_detected so cost triage can tell a runaway loop apart from a real crash. Adapter-agnostic — it observes the normalized tool-call stream, so it covers every CLI.

Rate-card snapshotting

Every metered row writes the rate_*_per_m columns from RateCard(provider, model) at the moment of write. When the rate card later changes, historical rollups stay consistent: the old row continues to imply its old rate. There are now two ways it can change — an edit to pricing.go (a provider repriced, a new model added), and a refresh of the embedded models.dev snapshot (go generate ./internal/modelcatalog/...), which can reprice any model that resolves at step 3 of the lookup below without anyone touching pricing.go. New rollup queries can either trust cost_usd directly or recompute from rate_*_per_m * tokens to verify nothing has drifted. Zero is the honest value for ollama/local (genuinely free) and for unknown-provider lookups (we couldn’t price). The columns are stored as nullable REAL (the v62 migration adds them without a NOT NULL / DEFAULT constraint), but the writer always supplies an explicit zero in those cases, so historical rows never observe NULL in practice — readers can treat the column as effectively populated.

Pricing — current rates

The canonical rate card lives in internal/paymaster/pricing.go. Adding a model is a one-line change. Prices are USD per 1,000,000 tokens and were verified against provider docs on 2026-04-30. It is no longer the only rate source: an embedded models.dev snapshot (internal/modelcatalog, 8 providers) sits underneath it and prices models the table has never heard of. The table stays authoritative — see The five-step lookup — and crewship model price --provider <p> --model <m> reports which source answered for any pair.
Anthropic Opus 4.7 was repriced to $5/$25 from the old $15/$75 in early 2026. The Crewship rate card was updated on 2026-04-30; ledger rows written before that date carry the old rate as a snapshot in rate_*_per_m and will continue to read out at the historical price. New rows use the corrected rate.

The five-step lookup

priceTable is no longer the only rate source. lookupPrice resolves (provider, model) in this order and stops at the first hit: The ceiling (step 4) picks the most-expensive known tier rather than the median, so budgets warn correctly even on premium reasoning models, at the cost of a mild over-estimate for cheap unknown ones. Better to over-estimate than to silently bill $0. The snapshot sits third on purpose: after the exact match, because priceTable carries verified corrections a bulk import must never overwrite (Opus 4.7 was billed 3× over until someone checked); and after the wildcard, so a future refresh that pulled an ollama or local provider into the trim could not start billing local calls. Models the snapshot carries no cost for — including 23 all-zero cost blocks, which are upstream gaps rather than free models — are skipped, never written as a $0 row, and fall through to the ceiling.
Tiered models are priced at their ceiling. 76 snapshot models publish a second rate card above a context threshold (typically 200k/272k, up to 6.7× base). Estimate has no context-size parameter, so the catalog flattens each model to its most expensive card via modelcatalog.Model.CeilingRates(). A short call on a tiered model is therefore over-estimated. The alternative — the base rate — would bill a long-context call at 15% of the invoice, and under-billing weakens the budget signal exactly when it matters. Selecting the right tier per call means changing Estimate, RateCard and lookupPrice together, which is a deliberate change, not a side effect. See Multi-provider LLM configuration.

The call path

paymaster.Middleware(next, j, db) wraps an LLMCaller. Order of operations:
1

Enforce

Check resolves every applicable budget; a hard/tiered breach short-circuits with *BudgetExceededError. The underlying call is never made, no ledger row is written, but budget.exceeded lands in the journal.
2

Call

Delegate to the inner caller.
3

Record

Fill cost_usd via Estimate(provider, model, tokens) if the provider didn’t price the call inline, then INSERT the row and emit llm.call + cost.incurred.
Billing errors do NOT fail the call: if the provider succeeded the response is returned unchanged and the operator sees the audit gap in logs. If the call failed, Paymaster still attempts a partial-billing record so the trail isn’t lost. See llm.middleware.go for the full stack (telemetry -> paymaster -> lookout -> raw).

Read endpoints

  • GET /api/v1/paymaster/spend/by-crew?range=7d — one row per crew (metered only).
  • GET /api/v1/paymaster/spend/by-agent/{crewId}?range=24h — per-agent rollup inside a crew (metered only).
  • GET /api/v1/paymaster/spend/by-mission/{missionId} — single-mission total (metered only).
  • GET /api/v1/paymaster/top-spenders?limit=10&range=7d — highest-cost scopes (metered only).
  • GET /api/v1/paymaster/subscriptions?range=30d — flat-rate credentials grouped by (plan, provider) with call count, token count, last-used timestamp. No $ figure — see “Billing modes” above for why.
Window syntax: range=1h|24h|7d|30d or explicit since=<RFC3339>&until=<RFC3339>. Default is 7 days. The four spend/* endpoints filter WHERE billing_mode = 'metered' so flat-rate rows do not silently inflate metered totals. The Subscriptions panel is the only place flat-rate rows surface to the operator.

Internal write endpoint

The sidecar writes ledger rows over the IPC socket, never directly. The handler is POST /api/v1/internal/cost/record (auth: X-Internal-Token). Authoritative scope (workspace / crew / agent IDs) comes from the sidecar’s IPCConfig set at exec time, so an agent that captured the token still cannot forge cross-tenant attribution. Mirrors the /internal/journal/emit security model — see Internal IPC API. Workspace isolation: the by-agent and by-mission handlers reject cross-tenant IDs with 404 (same shape as “not found”) so existence isn’t leaked. See crewBelongsToWorkspace / missionBelongsToWorkspace in internal/api/paymaster_handler.go.

CLI

Full reference: crewship paymaster.

Creating a budget

Budgets are inserted directly into budget_limits today — there is no dedicated API. The UI’s settings panel drives the same table.
Example workspace-wide hard cap of $250/day:

Sidecar coverage of agent CLI calls

Paymaster wraps llm.Provider.Complete() for everything called from the Go side (summaries, Keeper, consolidation, quartermaster judge). For agent CLI calls (claude, gemini, cursor-agent, droid) the sidecar is the metering point: the proxy intercepts the outbound HTTPS request, parses the response body for usage tokens, harvests rate-limit headers, and async-POSTs a Call to POST /api/v1/internal/cost/record. The handler validates auth and calls Record + EnforceQuota. What the sidecar can and cannot see: For pinned-cert tunnels the sidecar emits a “credential was used” attribution row (billing_mode=flat_rate, no tokens, cost_confidence=unknown) so the Subscriptions panel still shows usage. We deliberately do not MITM with a custom CA — CLIs with pinned certs would reject it, and the honest answer is that the body is private. Body observation uses io.TeeReader with a 10 MB cap so streaming UX is preserved while a pathological upstream cannot OOM the sidecar. Both JSON and SSE responses continue to pass through after the cap; only bytes beyond it are omitted from usage parsing. Look for the llm.call journal entry: if it exists, the call was metered. If you see exec.command with claude and no llm.call, the OAuth tunnel was used and the row will land as flat-rate.

Gotchas

  • Soft budgets still emit entries. A soft budget at 120% still writes budget.warning to the journal — the UI uses this to paint red. Don’t filter out “soft” when looking for “who was over”.
  • Stream path is unmetered. wrappedProvider.Stream() bypasses the full middleware (token counts arrive in the terminal message_delta event, which the sync CallResponse shape can’t carry). Streaming callers pay through orchestrator-level accounting that predates this package.
  • Pricing is estimated, and a “price” may be a ceiling. A new model from a provider the snapshot covers (anthropic, openai, google, deepseek, mistral, xai, openrouter, amazon-bedrock) is priced automatically at step 3. Anything else lands on the per-provider ceiling at step 4 — a real number, deliberately high, and not the invoice. The estimate does not fall to $0 unless the provider id itself is unknown, which is the case worth chasing: a registry row or a Provider.Name() with no rate row and no catalog entry bills every call at $0. Run crewship model price on the pair before assuming; pricing.go is where a correction goes, but it is no longer the first place to look.