Skip to main content

​
Feedback

Feedback is Crewship’s structured signal layer: helpful, not_helpful, “inaccurate”, “unsafe”, “edit”, “regenerate”. It sits alongside open-vocabulary emoji reactions but exposes a tight six-value enum so the eval pipeline and the online sampler can query a stable target without LIKE-matching codepoints.

​
Why feedback (and not reactions)

Reactions are social — 👍 on a message means “I liked this,” same UX as Slack. The eval pipeline can’t reliably mine that signal across emoji families (👍 vs 👍🏼 vs ☹️ vs 😀), and it can’t capture the strongest training signal of all: a user editing the assistant’s answer to what they actually wanted. The message_feedback table (migration v96) gives consumers a typed signal column with a CHECK constraint, a free-form reason, and a trace_id link back to the OTel trace of the run that produced the message. Eval datasets join on trace_id to recover the (prompt, original answer, preferred answer, ts) tuple.

​
Signal vocabulary

Adding a new signal requires both a UI patch and a migration that widens the v96 CHECK clause — the handler validates the enum before any DB work so a renamed value fails fast with a readable 400 instead of a SQLite constraint violation.

​
Privacy contract

Feedback is private to the author. Even a workspace owner cannot list another member’s thumb-downs or “edit” reasons — same threat model as a Slack reaction or a Google Docs comment, where candid signal needs a closed loop with the eval pipeline (server-side, not API-exposed). The UI’s optimistic-update store (stores/feedback-store.ts) persists per-user state to localStorage and reconciles via the API on submit. A 4xx/5xx response or transport rejection rolls back the optimistic flip so the local state never lies about a row that doesn’t exist on the server.

​
Trace correlation

Every POST accepts an optional trace_id. The intended end state is: the orchestrator stamps the active OTel trace id onto each assistant-message WebSocket event, the frontend lifts it onto ChatTurn.metadata.trace_id, and the feedback store passes it into the POST payload — so each feedback row lands indexed for WHERE trace_id = ? queries:
That powers the “show me every signal for this routine run” eval-mining query: collector trace ID → message_feedback.trace_id → all signals filed against that conversation.
End-to-end as of PR #450. The orchestrator → WebSocket → ChatTurn propagation is wired: internal/chatbridge/bridge.go calls telemetry.ResolveTrace(ctx) and stamps trace_id onto the "done" event metadata; hooks/use-chat.ts handleDoneEvent lifts it onto ChatTurn.metadata.trace_id; the feedback POST consumes it. New feedback rows land with trace_id populated whenever an OTel provider is configured. When no telemetry provider is configured (SpanContextFromContext returns invalid), the field is omitted — rows still land, just without the trace anchor — and the partial index WHERE trace_id IS NOT NULL keeps the lookup path cheap either way.

​
UI flow

The chat UI surfaces the signals through <TurnFeedbackActions> in components/features/chat/assistant-turn.tsx. Behaviour:
  1. User clicks thumb → optimistic state flip in the zustand store → background POST /api/v1/feedback with message_id, chat_id, trace_id, signal.
  2. On res.ok → state is the truth. On 4xx/5xx or network reject → roll back; user can retry.
  3. User clicks the same thumb again → DELETE /api/v1/feedback?message_id=...&signal=... first, then clear local state on success. A failed delete keeps the local state pointing at “submitted” so a refresh reconciles back to truth.
The store is intentionally async because syncing thumbs through the streaming useChat would couple two unrelated concerns. The trade-off is a one-frame flicker on flaky networks; the upside is that the chat path stays unaware of feedback wiring.

​
API contract

See the Feedback API reference for the full endpoint catalog. Quick sketch:

​
Workspace resolution

message_id is the only signal the server trusts for authorization (#1208, closed in #1213). On every POST the handler looks the id up in conversation_messages — the cross-session search mirror (migration v111) that internal/conversation.Store.Append writes on every real chat turn — reads that row’s own session_id, and confirms the caller is a member of the workspace that chat belongs to. A message_id the mirror has never seen, or one whose chat the caller can’t see, both collapse to the same 404 from the caller’s point of view; the two aren’t distinguishable from outside. chat_id is still accepted in the request body for older clients but is no longer trusted for authorization — a caller can no longer attach feedback to an unrelated message by pairing a real, visible chat_id with a forged message_id. There is also no more “fall back to the caller’s most-recent workspace” path: every row’s workspace_id comes from the message it’s attached to, every time, including on a re-POST:

​
Limits

​
Limitations

  • No server-side hydration of the UI state on initial load — the frontend reads localStorage; a fresh browser starts with empty state. The server is the truth, and GET /api/v1/feedback?message_id=... exposes it for any UI that wants to rehydrate.
  • No “retracted” signal value — DELETE removes the row entirely. The eval pipeline distinguishes “row exists” from “row doesn’t exist,” so a once-thumbed-then-untoggled message reads as “no signal” rather than “explicitly retracted.” Future iterations may add a retracted value if the distinction starts mattering.
  • message_id must be in the search mirror — the existence check (above) reads conversation_messages, which is “search-from-now-on”: messages from before migration v111 were never backfilled into it. A very old message_id can 404 here even though the JSONL transcript (chats.jsonl_path) still has it. Rare in practice, and visible immediately to the caller as a 404 rather than silently misattributing the row.