Skip to main content

​
Keeping crew images current

A crew container is created once and then reused for as long as the crew is alive. Crewship resolves and verifies the image digest at create time — it pulls by digest and records which manifest actually ran — but nothing looks again afterwards. On a self-hosted instance where crews stay up for weeks, that means a container quietly becomes a snapshot of whatever the registry held on the day it started. Nothing is broken while that is true. The container runs exactly the image it was built from. It is simply not the image the tag names any more, so security fixes and agent-runtime improvements published since have not reached it. Crewship now checks for this daily, notifies you, and gives you one command to act on it.

​
What “behind” means here

Two different staleness conditions exist, and they are deliberately kept apart. They are separate categories on purpose: someone who mutes agent errors to stop run-failure noise should not thereby lose “every crew on this instance is three releases old”, and someone watching instance hygiene should not be paged for one agent’s degraded container. The remediations differ too — a stale sidecar needs the binary rebuilt and recopied, a stale image needs a pull.

​
Checking one crew

This is a pure read: it never pulls and never restarts anything.
Status: current and Status: unknown are different answers. When the registry cannot be reached, or the crew runs a locally built crewship-cache:* image with no registry digest, there is nothing to compare — the command says unknown (registry unreachable) rather than pretending the crew is up to date. A freshness check that reports “current” when it could not look is worse than no check at all.
The reasons you may see:

​
Fixing it

Pulls the crew’s configured image, then force-removes its runtime container. The next agent exec recreates the container from the freshly pulled image. The pull happens first. If it fails — a rate-limited registry is the usual cause — the crew is left exactly as it was, rather than with neither a container nor a current image.
Agents executing in the container when it is removed are interrupted. On a busy crew, refresh when the crew is idle.

​
refresh-image vs restart-agents

Both drop the container. Only one pulls.
  • crewship crew restart-agents drops the container without pulling. Use it to pick up a devcontainer image you have already rebuilt, or an env/prompt change that only applies at container create.
  • crewship crew refresh-image pulls first. Use it when the base image has moved on in the registry — which is what image-status reports and what the notification is about.

​
The daily check

A platform routine (image_freshness) runs at 04:00 UTC on the same scheduler as the Keeper sweeps. It walks every live crew, asks the container provider for its image state, and journals an image.stale entry for each crew that is behind. From there the journal→notify bridge delivers it to whichever channels have Instance health enabled — see Notifications. The sweep registers only when the container provider can report image digests (the Docker provider can; the Apple-container provider cannot yet) and when a journal is wired. Otherwise it is skipped with an info line at boot.

​
How noisy is it

Not very, by construction. Two mechanisms:
  • Nothing unknown ever alerts. Every classification above that is not a confirmed digest mismatch reports “not behind”. An air-gapped instance, or one whose crews all run locally built cache images, produces zero notifications forever.
  • One alert per observation, not per day. Before emitting, the sweep checks the journal for an existing image.stale entry with the same (crew, running digest, resolved digest). A crew nobody has recycled stays quiet after the first notice. When the tag moves again, that is a new pair and a new notice — and a crew that has been refreshed and later falls behind once more alerts properly rather than being silenced forever.
The de-duplication lives in the journal rather than in memory precisely so that a restart or a redeploy does not re-alert on a condition nobody has touched.
The check is per crew, not per agent run. Agents in the same crew share one container, so “this crew is behind” is the honest unit — and checking on the dispatch path would put a registry round-trip on every run.

​
Seeing it in the UI

Open a crew, go to Settings → Runtime & security → Container image & features. The Image freshness card reports the same three-way answer as the CLI — current, behind, or unknown-with-a-reason — with a Refresh image button when the crew is behind. Both digests are shown, so you can see what moved.

​
Turning the notification on

Instance health is a standard notification category. In Settings → Notifications, enable System → Instance health on the channel you want. Nothing else is required; the category already exists and the sweep is its producer.
Before this existed, system.health was the one category in the matrix with no journal producer — a switch you could turn on that could never deliver anything. The image-freshness sweep is what it now delivers.