Harbormaster

Harbormaster is the HITL approval workflow. Agents call Gate before high-risk actions (destructive ops, production targets, expensive tool calls); if a rule matches, the action is queued in approvals_queue and either logged-and-continued (async) or paused until a human decides (sync). Decisions and timeouts emit journal entries so the audit trail is complete. This complements Lookout: Lookout sanitises content; Harbormaster gates actions.

Gate modes

const (
    ModeNone  Mode = iota // bypass rules entirely (trusted callers)
    ModeAsync             // enqueue, agent keeps going
    ModeSync              // enqueue, block agent until decided or timeout
)

Mode	Behaviour	Use
`ModeNone`	Returns `{Approved: true, NotGated: true}` without consulting rules.	Keeper or system-level callers that are trusted.
`ModeAsync`	If a rule matches, enqueue and return `Pending=true`. Caller continues.	Low-risk logging; humans see a trailing record but the agent never waits.
`ModeSync`	Poll `approvals_queue` every second until status leaves `pending`.	High-risk actions that must not proceed without a human yes/no.

Sync mode respects ctx.Done() (so request cancellation unblocks the poll) and uses a client-side deadline (TimeoutSecs, default 3600) on top of the server-side timeout sweeper — either can flip the row to timeout and unblock the gate.

Rule evaluation

The default Evaluator comes pre-loaded with rules for destructive ops, cost thresholds, and production target patterns (see rules.go). Callers can compose their own:

eval := harbormaster.NewEvaluator(
    harbormaster.RuleMatcher{
        Name:        "destructive_shell",
        ToolPattern: regexp.MustCompile(`(?i)^(rm|drop|delete|truncate)`),
        MapsToKind:  harbormaster.KindDestructiveOp,
    },
    harbormaster.RuleMatcher{
        Name:             "expensive_tool",
        CostThresholdUSD: 5.0,
        MapsToKind:       harbormaster.KindCostThreshold,
    },
    harbormaster.RuleMatcher{
        Name:              "production_target",
        TargetEnvPatterns: []string{"prod", "production"},
        MapsToKind:        harbormaster.KindTargetEnvironment,
    },
)

A rule fires when ANY of its non-zero conditions match. RequireWhen(tool, args) is a free-form last-resort predicate. The orchestrator wires this via approvalGateAdapter in internal/server/orchestrator_adapters.go, which uses NewEvaluatorWithDefaults().

Where the run-level mode comes from

Before a run starts, the orchestrator calls the gate once with tool agent_run. The mode for that check — none, async, or sync — is derived from the run’s crew autonomy level and stamped onto the dispatch request by the single request-builder that every path (chat, pipeline, cron, webhook, mission, peer) funnels through. The mapping is:

Autonomy level	Gate mode	Meaning
`strict`	`sync`	Every gated action blocks for a human decision.
`guided`	`sync`	Writes need an OK (default for new crews).
`trusted`	`async`	Gated actions log to the inbox; the agent keeps going.
`full`	`none`	Autonomous — the gate is bypassed, journal-only.

Because the run-level mode is now sourced from policy on every dispatch path, the gate is live everywhere, not just interactive chat. Note that the baked-in default rule set does not match the agent_run tool, so raising a crew to sync does not block ordinary runs on its own — it takes effect once a rule matches the run (custom agent_run rules, cost, or a production target). The mode governs whether the rules are consulted at all.

Queue schema

CREATE TABLE approvals_queue (
  id               TEXT PRIMARY KEY,
  workspace_id     TEXT NOT NULL,
  crew_id          TEXT,
  agent_id         TEXT,
  mission_id       TEXT,
  requested_by     TEXT,
  kind             TEXT NOT NULL,  -- tool_call | cost_threshold | destructive_op | target_environment | custom
  reason           TEXT,
  payload          TEXT NOT NULL DEFAULT '{}',
  status           TEXT NOT NULL CHECK (status IN ('pending','approved','denied','timeout','cancelled')),
  decided_by       TEXT,
  decided_at       TEXT,
  decision_comment TEXT,
  timeout_at       TEXT,
  created_at       TEXT NOT NULL
);

Endpoints

GET /api/v1/approvals?status=pending&limit=50 — inbox. Status defaults to pending; use ?status=all for full history.
GET /api/v1/approvals/{id} — full request including payload.
POST /api/v1/approvals/{id}/decide — body {"status":"approved|denied","comment":"..."}. Requires OWNER or ADMIN workspace role; 403 otherwise.
POST /api/v1/approvals/{id}/cancel — withdraw a still-pending request without approving or denying it (flips the row to cancelled, records no approve/deny decision). Use when the request is moot.
POST /api/v1/approvals/reset-auto-tuning — body {"tool":"..."}. Clears the rolling gate_reward_history window for one tool (see Reward-adjusted gating).

Cross-tenant IDs return 404 with the “not found” shape. Deciding a non-pending row returns 409 (already decided). See Approvals API for full schemas.

Timeout sweeper

harbormaster.StartTimeoutSweeper(ctx, db, j, 30*time.Second) runs a background goroutine that flips rows past timeout_at from pending to timeout and emits approval.timeout. The server starts this once at boot. On ModeSync, the sweeper AND the client-side deadline both try to flip the row — whichever wins, the row is consistent. A race with a last-second decide is handled: if the UPDATE affects zero rows Gate re-reads the row and returns the human decision rather than misreporting a timeout.

Standing approvals leave a trail

A routine trust grant disarms a wait:approval gate for one step of one routine body — the “you have approved this same gate twelve times, stop asking me” surface. It is the bluntest decision in the HITL system, because it removes the human from the loop rather than answering one question in it. Granting and revoking therefore emit journal entries with the same shape every other decision uses (harbormaster.AfterDecide):

Entry type	When	Actor
`approval.trust_granted`	a gate stopped asking	the granting user
`approval.trust_revoked`	a gate starts asking again	the revoking user

Both carry refs.trust_grant_id, refs.pipeline_slug, refs.step_id and payload.definition_hash. The hash is the load-bearing field: a grant only fires against that exact routine body, so an entry without it records that something was trusted but not what. Filter with approval.* to see one-off decisions and standing ones together — they are the same control at two levels, which is why they share a namespace rather than getting a trust.* family of their own. A revoke that changed no row (already revoked, wrong routine) emits nothing. An entry for a withdrawal that did not happen would make the journal disagree with the table, which is worse than an entry that is missing.

Escalations: the other human decision

Harbormaster gates an action an agent is about to take. An escalation is the reverse direction — an agent asking a human a question and blocking on the answer (internal/api/escalation_handler.go, raised through the sidecar’s /escalate). Both are human decisions, and until release 1.0 only one of them could end.

The state machine

PENDING → RESOLVED    a human decided; `action` is approve | reject | redirect
        → EXPIRED     the deadline passed and no human decided
        → CANCELLED   a human withdrew the question before deciding

Terminal is terminal: resolving or cancelling a row that already reached one of the three returns 409, with a message naming the state it actually reached. RESOLVED was kept rather than renamed to ANSWERED, and rejection stays in the action column rather than becoming a fourth status — see internal/database/migrations/20260813212851_escalation_deadline.sql for why either change would have been a second spelling of “done”.

Two clocks: the agent’s wait and the human’s answer

An escalation is bounded twice, because it contains two different questions.

Column	Bounds	Default	What running out means
`deadline_at`	the agent’s long poll	300 s	the agent stops waiting and continues with a warning; `agent_gave_up_at` is stamped. The row stays `PENDING`.
`answer_deadline_at`	the human’s answerability	7 days	the row goes `EXPIRED`, and any staged credential is disposed of

The create response publishes deadline_at + timeout_seconds (the agent’s window — the sidecar bounds its long poll on what the server told it, not on a constant of its own) and answer_deadline_at separately, for the console and the CLI.

An earlier build of this feature wrote one deadline and used it for both. The approval queue then expired 300 s after the agent asked: an operator saw the inbox item, walked off to fetch the API key, came back seven minutes later, clicked Approve — and got 409 expired for a question nobody had ever answered. Do not collapse these two numbers. They measure a poll and a person.

The agent’s clock is still server-owned, which was the real fix in the first version and is kept: the agent used to give up after a hardcoded 300 s while the row stayed PENDING forever because nothing told the server the question had been abandoned. Now the long poll ends on deadline_at, and the same event stamps agent_gave_up_at — a fact about the run, not a decision about the question. Expiry — the human’s clock — is swept, exactly as approval timeouts are: QueryHandler.StartEscalationExpirySweeper runs every 60 s, and every escalation read path (escalation list, pending-count) sweeps its own workspace first so no surface can show a question as open past the point where it can still be answered. Every path goes through one compare-and-swap on status = 'PENDING', so the transition — and its journal entry — happens exactly once no matter how many observers notice at the same moment. Losing that CAS at the deadline means a human decided in the same instant, and the waiter returns their decision rather than a false expiry. Rows with a NULL answer_deadline_at (raised before the column existed) never expire. Back-filling them would retro-expire questions somebody may still intend to answer. One knock-on worth knowing: PENDING rows now persist for days rather than minutes, so a crew’s routine-escalation backlog budget (escalation.max_pending_per_crew, default 10) is consumed until an operator actually answers. That is what that cap was designed against; the five-minute window had been quietly refunding it.

What happens when nobody answers

The agent continues, with an explicit warning. The run is not failed, it is not paused, and no default answer is invented.

This was previously decided by omission — the client timed out and the agent proceeded in silence. It is now a stated policy, with two distinct wire answers because there are two distinct situations:

Wire `status`	When	Row
`UNANSWERED`	the agent’s window closed, the question is still open	`PENDING`, `still_open: true`
`EXPIRED`	the answer deadline passed	terminal, `resolved_by = 'system'`
`CANCELLED`	a human withdrew the question	terminal, `resolved_by = 'user'`

All three carry an empty resolution and a mandatory warning telling the agent to continue without the answer, not to assume approval, to say in its result that the question went unanswered, and to avoid irreversible actions that depended on it. UNANSWERED adds one more instruction — do not wait for it and do not ask again — because an agent told “no answer yet” would otherwise reasonably infer that it should keep waiting, which is the wait that just ended. UNANSWERED is a wire status only. It is deliberately not a row status: nothing transitioned, so nothing claims a transition, and an agent handed EXPIRED for a question an operator can still see and answer in their inbox would be reporting a state the system is not in. An expiry writes a peer.escalation journal entry with payload.state = "expired", payload.agent_outcome = "continued_with_warning" and severity warn, so it surfaces in the default “needs attention” filter. An operator finds out that a decision they meant to make was made for them by a clock. Failing the run instead was considered and rejected: an agent blocked on a question can usually make progress without it, and failing runs because a human was at lunch makes the escalation tool too expensive for agents to use. What is not acceptable is doing any of that silently. The same warning discipline covers the degraded paths. If the sidecar loses the connection or gets a 5xx it answers status: TIMEOUT — it does not know the outcome, so it does not claim one — but it still carries a warning. {"status":"TIMEOUT","resolution":""} with nothing else reads to a model exactly like a question that was answered with silence.

Answering after the agent gave up

This is the case the single-deadline design avoided by making everything expire at once, and it is now explicit. Resolving a PENDING escalation whose agent_gave_up_at is set succeeds — 200, not 409 — and the response says what it did and did not accomplish:

{
  "id": "esc_abc",
  "status": "RESOLVED",
  "action": "approve",
  "agent_still_waiting": false,
  "agent_gave_up_at": "2026-08-18T09:14:02Z",
  "note": "Recorded, but nela stopped waiting at 2026-08-18T09:14:02Z and continued without this answer — that run will not receive it. The credential is now active in the vault, so the next run has it."
}

The run that asked is not resumed. It was told in writing, minutes or days earlier, that it was continuing without the answer, and it has since finished; there is nothing left to deliver to, and reviving it would be a second, unannounced execution of work an operator already saw complete. What a late answer does accomplish is why it is still worth giving:

a CREDENTIAL approval activates the staged credential, so the next run has the secret and the agent does not have to ask again — which is exactly what the operator walking to the password manager was trying to achieve;
the decision is on the record, journaled with payload.agent_still_waiting: false, so “why did that agent proceed without my approval” is answerable from the resolution entry rather than only from a give-up hours earlier.

The console and the CLI get agent_gave_up_at on every listed escalation for the same reason. A surface that shows the agent’s deadline as the operator’s countdown recreates the bug in the UI after it was fixed in the server.

Cancelling vs rejecting

POST /api/v1/escalations/{id}/cancel withdraws a question that stopped mattering. It is deliberately not resolve --action reject: a rejection is a decision the agent should act on (“no, do not do that”), while a cancellation says nobody ever considered it. Collapsing them would tell an agent it was refused when it was not. Cancellation is MANAGER+, journaled with the operator’s user id and reason, and unblocks any waiting agent with the same “no answer” warning.

Staged credentials do not outlive their question

A CREDENTIAL escalation may carry an agent’s proposal: the value is encrypted into the vault up front as a PENDING_APPROVAL credential, and approving the escalation is one click rather than a human retyping a secret. That row is reachable through exactly one route — the escalation’s resolve path (approve → ACTIVE, reject → REJECTED + deleted_at). So every terminal transition has to dispose of it, not just rejection. Expiry and cancellation now do, with the same REJECTED + deleted_at disposal and a REJECTED credential-audit event carrying disposed_reason. Before that, an expired or cancelled credential escalation left an encrypted secret that no route could activate and no route could reject, while the name-conflict probe counted it as a live name — so every later proposal of that name came back as a conflict and the agent was told to have a human type the value in by hand. One unanswered question jammed auto-staging for that name permanently. The probe itself is now keyed on reachability rather than on the row merely existing: a PENDING_APPROVAL credential conflicts only while a PENDING escalation still links it. A proposal whose question is still open genuinely is live and must still conflict; a proposal whose question is terminal is dead and is retired in passing.

CLI

crewship approvals list                             # inbox
crewship approvals list --status approved --limit 100
crewship approvals approve <id> --comment "looks safe"
crewship approvals deny <id>   --comment "wrong mission"
crewship approvals cancel <id> --comment "request is moot"   # withdraw, no decision

See crewship approvals for the full command surface. Pending requests resolve by approve, deny, cancel, or the server-side timeout sweeper. The escalation side has the mirror set:

crewship escalation list --crew backend --status PENDING     # DEADLINE column shows the agent's wait window
crewship escalation resolve <id> --action approve --resolution "yes"
crewship escalation cancel  <id> --reason "the deploy was rolled back"   # withdraw, no decision
crewship escalation sweep-expired                            # force the deadline sweep now

See crewship escalation.

Hook integration

The hooks system fires on_approval_requested when Harbormaster determines approval is required — specifically, when Gate returns Required=true (applies to Approved, Denied, and Pending branches). The orchestrator’s HookDispatcher dispatches the event after the gate decision lands; the harbormaster package itself stays hook-agnostic. Use this hook to page oncall, post to Slack, or auto-escalate:

# hooks_config row (illustrative -- registration is config-time)
event: on_approval_requested
handler_kind: http
handler_config:
  url: https://hooks.slack.com/services/...
  method: POST
matcher:
  severities: ["high", "critical"]

Reward-adjusted gating

Every Decide call also feeds gate_reward_history — one row per outcome, keyed by (workspace_id, tool_name, args_hash). On the next Gate() call for the same shape, harbormaster.AdjustMode walks the last 20 outcomes and:

downgrades sync → async when approval rate > 90% (humans are rubber-stamping — stop blocking the agent)
upgrades async → sync when denial rate > 70% (humans are rejecting — start blocking instead of logging and running anyway)

Both require a quorum of at least RewardHistorySize/2 = 10 decisions before tuning — a single denial won’t flip the mode. Timeouts and cancellations are tracked but excluded from the rate calculation so inaction doesn’t dilute operator intent. Every mode change emits a keeper.rule_auto_tuned journal entry so the audit trail shows why a later call took a different path than the rule says. Reset — operators can wipe the rolling window for a tool via CLI:

crewship approvals reset-auto-tuning shell.exec
# → Reset auto-tuning for "shell.exec" — cleared 17 rows from gate_reward_history

or HTTP:

curl -X POST /api/v1/approvals/reset-auto-tuning -d '{"tool":"shell.exec"}'

Use when automation approved on behalf of humans for a while and biased the window (the next decisions will retrain naturally). args_hash is a sha256 over JSON-sorted keys — the raw args are never stored in gate_reward_history, only in the original approvals_queue row. Semantically-equal calls hash the same, so one cohort per operation shape. Inspired by Self-Evolve’s Q-value update loop, simplified: operator decision is the signal (no LLM judge needed).

Gotchas

Edge cases and footguns

Only OWNER and ADMIN can decide. The Decide handler inline-checks RoleFromContext and returns 403 for anyone else. This used to be documented as “middleware-enforced” but there was no middleware — the check is now explicit in the handler.
Soft-delete = denial. If the row vanishes between enqueue and poll (e.g. DB cleanup), Gate fails closed with Denied=true.
Sync mode holds an HTTP goroutine. A long-running sync approval pins one connection. Don’t route high-volume traffic through sync mode; use async + a hook for routing.
TimeoutSecs is per-call. If a caller passes 30 and the sweeper interval is 30, you can get one extra poll where the row is still pending but timeout_at has passed — both paths converge, but test expectations should allow 1-2s of slop.

Crew Journal — approval.requested, approval.granted, approval.denied, approval.timeout, approval.cancelled, approval.trust_granted, approval.trust_revoked, and peer.escalation (with payload.state of pending | resolved | expired | cancelled).
Hooks — on_approval_requested event.
crewship approvals, Approvals API.
crewship escalation, Crews API — the escalation lifecycle.

Get Started

Guides

Security

Configuration

Manifest Reference

Harbormaster

Harbormaster

Gate modes

Rule evaluation

Where the run-level mode comes from

Queue schema

Endpoints

Timeout sweeper

Standing approvals leave a trail

Escalations: the other human decision

The state machine

Two clocks: the agent’s wait and the human’s answer

What happens when nobody answers

Answering after the agent gave up

Cancelling vs rejecting

Staged credentials do not outlive their question

CLI

Hook integration

Reward-adjusted gating

Gotchas

​Harbormaster

​Gate modes

​Rule evaluation

​Where the run-level mode comes from

​Queue schema

​Endpoints

​Timeout sweeper

​Standing approvals leave a trail

​Escalations: the other human decision

​The state machine

​Two clocks: the agent’s wait and the human’s answer

​What happens when nobody answers

​Answering after the agent gave up

​Cancelling vs rejecting

​Staged credentials do not outlive their question

​CLI

​Hook integration

​Reward-adjusted gating

​Gotchas

​Related

Harbormaster

Gate modes

Rule evaluation

Where the run-level mode comes from

Queue schema

Endpoints

Timeout sweeper

Standing approvals leave a trail

Escalations: the other human decision

The state machine

Two clocks: the agent’s wait and the human’s answer

What happens when nobody answers

Answering after the agent gave up

Cancelling vs rejecting

Staged credentials do not outlive their question

CLI

Hook integration

Reward-adjusted gating

Gotchas

Related