Skip to main content

Consolidate

Two background workers run against the Crew Journal. Both start automatically at server boot (consolidate.StartBackground) on the cadences below; you can also trigger a consolidation run on demand via the API/CLI.
  1. Consolidator — runs every 6h. Walks recent journal entries per crew and produces a set of learned rules appended to .memory/topics/learned-YYYY-MM-DD.md. Emits memory.consolidated.
  2. Compactor — runs daily at 03:00 UTC. Rolls up low-signal entries older than 30 days into a single system.compaction entry and deletes the originals. warn/error rows are preserved indefinitely.
Each degrades gracefully when its inputs are absent: no summarizer (auxiliary LLM) configured → the consolidator loop still runs but produces no rules; the compactor always runs its retention sweep.
The “supervisor crew” orchestration pattern (a dedicated crew that reviews and curates consolidated rules) is on the roadmap and not part of the current automatic workers.

Candidate entry types

The consolidator considers these entry types for rule extraction:
Everything else (exec chunks, metrics, broadcast noise) is explicitly excluded so the LLM prompt stays dense and the rules grounded in decisions rather than activity.

Rules output

Rules land in {memoryRoot}/{crewSlug}/topics/learned-YYYY-MM-DD.md. Format:
The Lead’s crew memory search (FTS5) picks these up automatically — see Memory System.

Summarizer

SummarizerClient is an interface:

Which model runs it

The summarizer resolves the curator auxiliary slot — the slot the Judge models card and crewship keeper aux list have always described as “Skill review + memory consolidation”. Repointing it applies to the next consolidation, not the next restart:
If that slot has no buildable provider — the common case being the shipped anthropic default on an instance with no ANTHROPIC_API_KEY — consolidation falls back to the boot-time KEEPER_OLLAMA_URL + KEEPER_MODEL client, so an install running a local judge with no API key keeps consolidating. Only when there is neither is the summarizer absent. The curator slot’s --timeout bounds the slot’s evaluator (skill review) and the operator-model sweep; it is deliberately not applied to a consolidation prompt, which is batch work over up to a few hundred journal entries and is bounded by the provider’s own client timeout instead. If no summarizer is configured at all:
  • Background runner skips every crew silently (the pin-snapshot path still runs).
  • Manual trigger returns 202 Accepted with note: "no summarizer configured, skipping" and emits system.consolidation_triggered + system.consolidation_completed (status=skipped-no-summarizer) so the audit trail captures the no-op.

Compactor

Compaction target: info and notice entries older than 30 days that are NOT type summary.generated, memory.consolidated, or any checkpoint/fork/approval/regression. Rolls up into one system.compaction entry per (crew, day) bucket with a count and kind breakdown in the payload, then archives the originals to journal_entries_archived (with the payload truncated to 400 chars) before deleting from the base table. Done in a single transaction so neither side can drift. Preserved indefinitely (never compacted):
  • warn and error severity
  • summary.generated, memory.consolidated
  • Everything under approval.*, checkpoint.*, fork.*, eval.regression_detected
  • Anything tagged with priority='permanent' (see Crew Journal — Priority markers)

Archive layer (PR #212)

Migration 55 added journal_entries_archived so compaction is no longer destructive. The aged row is copied into the archive with its summary truncated to 200 chars and its payload truncated to 400 chars (stored in the compressed_payload column); the pre-truncation size is recorded in original_size_bytes. There is no trailing marker — the copy is a plain substr. The archive table has the same workspace/ts indexes as the base, so historical forensic queries still work — they just have to include the archive table:
The frontend /journal view does not read from the archive by default — operators have to explicitly opt in via ?include_archive=true. This keeps the live UI fast and the archive as a forensic-only surface. If you want to tune journal compaction retention, edit RunnerOptions.CompactionOlderThan and restart. Per-workspace retention for memory_versions (a separate Iter 4 worker sweep) is configurable at runtime via PATCH /api/v1/admin/memory/config — see Memory Observability — Per-workspace retention.

Manual trigger

OWNER or ADMIN only. Returns 202 Accepted immediately; the actual run happens in a 10-minute background goroutine. Response shapes: since accepts Go durations (90m, 24h) plus shorthand d (days) and w (weeks). Default 24h if unset or unparseable. Soft-deleted crews are treated as not found — a deleted crew can’t grow fresh memory artifacts via explicit ID.

CLI

Full reference: crewship consolidate.

Journal entries emitted

Gotchas

  • Ollama must be reachable when the curator slot points at it (and for the episodic indexer, which always does). Set OLLAMA_MODELS="/Volumes/SSD 990 PRO/ollama-models" and ollama serve before starting the server locally. Consolidation itself is no longer Ollama-only: a curator slot on a hosted provider works with no Ollama at all.
  • The triggered journal entry is always emitted. Even the “no summarizer” skip path emits both triggered + completed so operators see the full round-trip. If you filter for memory.consolidated only, you’ll miss skipped runs.
  • Per-workspace in-flight guard. A second manual trigger for the same workspace while the first is running returns 409. Track worker_id if you need to wait for completion.
  • Compactor is aggressive on info/notice. If you rely on exec.command entries older than 30 days for forensics, export them out-of-band before the Compactor rolls them up.