Devcontainers & Runtime Images

Crewship used to ship a single monolithic ghcr.io/crewship-ai/agent-runtime image with every CLI, tool, and the sidecar baked in. That image has been retired (commit dd86356). Any glibc-based Linux base image now works — Crewship provisions the crew-specific tooling on top via devcontainer features and bind-mounts the sidecar from the host. This guide explains the whole pipeline end-to-end, from setting a base image on a crew to running agents inside the cached per-crew image.

What you get

Bring Your Own Image (BYOI)

Use debian:bookworm-slim, ubuntu:24.04, mcr.microsoft.com/devcontainers/base:bookworm, or any other glibc Linux image. Musl/Alpine is unsupported (the Go sidecar binary is CGO-free but still glibc-linked).

Community Features

Declarative tooling via the open devcontainer Features spec. Dependencies sort automatically (installsAfter).

mise Runtime Tools

Pin exact language versions (Node 22, Python 3.12, Terraform 1.9) via a mise.json block — no Dockerfile required.

Per-Crew Cached Image

Provisioning commits the container as crewship-cache:{hash[:12]}. Re-running agents skips the install phase entirely.

Mental model

+----------------------------------------------------------------+
|  Host (your Mac / VM / server)                                 |
|                                                                |
|  crewshipd    <---- HTTP/WS ----   crewship CLI & UI           |
|    |                                                           |
|    | provisioner.Provision()                                   |
|    v                                                           |
|  +----------------------------------+                          |
|  | 1. Pull base image               |  debian:bookworm-slim    |
|  | 2. Create temp container         |                          |
|  | 3. Install devcontainer features |  common-utils, claude... |
|  | 4. Install mise tools            |  node@22, python@3.12    |
|  | 5. Run postCreateCommand         |                          |
|  | 6. docker commit -> cache image  |  crewship-cache:a1b2c3…  |
|  +----------------------------------+                          |
|                                                                |
|  Per-crew runtime (EnsureCrewRuntime):                         |
|  +----------------------------------+                          |
|  | image = CachedImage || RuntimeImg|                          |
|  | Bind mounts (read-only):         |                          |
|  |   crewship-sidecar -> /usr/local/bin/crewship-sidecar       |
|  |   entrypoint.sh    -> /usr/local/bin/entrypoint.sh          |
|  | Bind mounts (rw):                |                          |
|  |   /workspace /output /crew /secrets                         |
|  | Volumes: /home/agent, /opt/crew-tools                       |
|  +----------------------------------+                          |
+----------------------------------------------------------------+

Phase 1: Dockerfile + BuildKit provisioning

As of Phase 1, when the Docker CLI with BuildKit is available the provisioner no longer builds the runtime by spinning up a temp container and docker commit-ing it. Instead it generates a Dockerfile per crew and builds it with BuildKit, which buys:

Per-feature layer caching — adding one feature rebuilds only that layer; common-utils and the previously-installed features hit the cache.
Package-manager cache mounts — apt / npm / pip caches persist across builds, so re-provisions don’t re-download the world.
Deterministic ordering — features install in dependency order (topological sort with common-utils first).

The build shells out to the docker CLI (that is how BuildKit’s cache mounts and the dockerfile frontend behave identically everywhere), and the CLI is pinned to the daemon the container provider resolved — not to whatever docker context happens to be current. This matters on any machine with more than one runtime installed: colima start and a Rancher Desktop launch both make themselves the current context without touching /var/run/docker.sock, so an unpinned build would land the image on one daemon while the container create looked for it on another. Your docker context is yours; it has no effect on where Crewship builds. Intermediate per-feature layers are tagged crewship-feat:{hash} (regenerable, never referenced directly); the final per-crew image is still tagged crewship-cache:{hash[:12]}. The transition is transparent: if BuildKit is unavailable on the host, provisioning falls back to the container-commit path shown above with no operator action required. Either way you trigger it with the same crewship crew provision / crewship crew rebuild commands.

Prerequisites

Build the sidecar on the host

The sidecar is a CGO-free Go binary that Crewship bind-mounts into every crew container. It is mandatory — there is no baked-in fallback anymore.

make build:sidecar
# Produces:
#   ./crewship-sidecar        (the Go binary)
#   ./entrypoint.sh           (init script, copied from scripts/entrypoint.sh)

Autodetect in internal/config/config.go searches:

{binDir}/crewship-sidecar (next to the crewship binary)
/usr/local/bin/crewship-sidecar

And for the entrypoint:

{binDir}/entrypoint.sh
{cwd}/scripts/entrypoint.sh
{cwd}/entrypoint.sh
/usr/local/share/crewship/entrypoint.sh

Override explicitly with CREWSHIP_SIDECAR_PATH / CREWSHIP_ENTRYPOINT_PATH.

Verify fail-fast is in effect

If neither file can be located, crewship start exits with:

sidecar binary not found (tried [/opt/crewship/crewship-sidecar /usr/local/bin/crewship-sidecar]);
run 'make build:sidecar' or set CREWSHIP_SIDECAR_PATH
(or CREWSHIP_SKIP_SIDECAR=1 to bypass in tests)

This replaces the old silent-fallback behaviour that would launch containers without a sidecar.

Confirm Docker can pull images

Provisioning runs docker pull against the base image you declare. On air-gapped hosts, pre-pull:

docker pull mcr.microsoft.com/devcontainers/base:bookworm

Configuration shape

Every crew stores three optional fields in the crews table:

Field	Type	Purpose
`runtime_image`	string	Base OCI reference; default `debian:bookworm-slim`. Seed demos use `mcr.microsoft.com/devcontainers/base:bookworm`.
`devcontainer_config`	JSON string	Subset of the devcontainer.json spec: `image`, `features`, `postCreateCommand`, `containerEnv`, `remoteUser`.
`mise_config`	JSON string	mise `config.toml` in JSON form: `{"tools":{"node":"22","python":"3.12"}}`.

Supported `devcontainer.json` fields

Parser lives in internal/devcontainer/config.go:

{
  "image": "mcr.microsoft.com/devcontainers/base:bookworm",
  "features": {
    "ghcr.io/devcontainers/features/common-utils:2": {
      "username": "agent",
      "userUid": "1001",
      "userGid": "1001",
      "installZsh": false
    },
    "ghcr.io/devcontainers-extra/features/claude-code:2": {},
    "ghcr.io/devcontainers/features/github-cli:1": {}
  },
  "postCreateCommand": "echo ready",
  "containerEnv": {
    "TZ": "UTC"
  },
  "remoteUser": "agent"
}

postCreateCommand accepts the polymorphic forms from the spec: a single string, an array of strings (run sequentially), or an object of named commands (keys ignored, values executed). See parsePolymorphicCommand in internal/devcontainer/config.go.

mise config

{
  "tools": {
    "node": "22",
    "python": "3.12",
    "terraform": "1.9"
  }
}

mise tools are installed as the agent user (UID 1001) under /opt/mise — data and shims in /opt/mise/data, the global config in /opt/mise/config/config.toml — and the runtime container carries the matching MISE_* variables (cache and state, the two mise writes to on every call, point under /home/agent, the writable volume: the container’s root filesystem is read-only). Not under /home/agent for the tools themselves: that directory is a per-crew named volume at runtime, and a volume hides whatever the image has at its mount point, so tools installed into the home inside the image were absent from the running container. /opt/mise is image-resident and survives cache rebuilds.

The agent’s PATH

The agent runs through a non-login docker exec, which does not read /etc/profile.d. The build therefore puts the agent’s own tool directories at the front of the image PATH — /home/agent/.local/bin, /opt/crewship/bin and /opt/mise/data/shims — in the image ENV, in /etc/environment and in the runtime container environment, so every mise tool and every agent-installed CLI resolves without a login shell. At container start the runtime merges three sources into one PATH — these directories, the build’s aggregated containerEnv PATH (feature-declared directories and the image’s own, with ${containerEnv:PATH} expanded), and the login PATH captured at build time — so nothing a feature or a profile script added is lost, and a captured login PATH that does not know about mise’s shims cannot hide them. You do not need to spell these out in containerEnv.PATH; a PATH you set there is kept behind them.

Adapter CLIs are installed and verified

Provisioning also records the required agent CLIs’ observed versions and paths inside the image. crewship crew provision status <slug> shows the built versions; JSON output separates toolchain.requested selectors from toolchain.built evidence and its immutable image ID. An old image, an unsupported version response, or an unavailable inspector is reported as unknown. This describes the built image, not a running agent’s writable files or a compatibility qualification for a particular model/account. Native Apple Containers currently bakes the inventory but cannot read it back through this inspector; Docker-backed installations can. Crewship disables Claude Code’s background updater with DISABLE_AUTOUPDATER=1 and Codex’s startup update check with check_for_update_on_startup=false. OpenCode receives OPENCODE_DISABLE_AUTOUPDATE=1. Gemini receives per-run system settings that disable automatic updates and update notifications, including the older inverted setting names. Failure to write those settings stops preflight. The Gemini and OpenCode controls were checked against Gemini v0.62.0 and OpenCode v1.18.34. Their current automatic updater call sites are interactive; these settings also cover interactive descendants inheriting the run environment. They do not stop an agent with a shell from deliberately installing or running another binary. Equivalent controls for Cursor and Droid have not been qualified. Updates belong to environment provisioning. An agent on CLAUDE_CODE needs claude in the image; CODEX_CLI needs codex, GEMINI_CLI gemini, OPENCODE opencode, CURSOR_CLI cursor-agent, FACTORY_DROID droid. Nobody has to remember that: every build reads the crew’s agents and adds what is missing — a mise tool for each CLI no declared feature already provides (claude, codex, gemini-cli, opencode, cursor-agent), an install command for droid. Your own choices are kept: a pinned claude version in the mise config, or the claude-code feature, is used as-is. After the install steps the build runs command -v for every required binary as the agent user, with the agent’s PATH. A binary that did not land fails the build with its name — adapter CLI not installed in this image: claude — instead of the first chat message failing with “No such file or directory”. The verified binaries are recorded with the image, and the dispatch gate reads them: before an agent runs — from chat, an issue, a routine, a container start — the crew’s image must be present and verified for the adapter of every live agent, or a rebuild is enqueued and the message waits for it. So an agent added by any path (the wizard, a hire, a template, onboarding, a manifest) is covered before it first runs, and a plain base image with no features still gets the build its agents need. A crew whose image was built before this verification existed rebuilds once. Creating or moving an agent onto an uncovered adapter enqueues the rebuild right away rather than at the first message. Binaries the build installs on the agent’s behalf that mise does not carry (the Factory droid CLI) land in /opt/crewship/bin, an image-resident directory owned by the agent user and on its PATH.

Managing a crew’s runtime config

UI (Crew Wizard)
CLI
REST API

The crew creation / edit wizard has a Runtime Configuration step with:

Base image dropdown (populated from /api/v1/runtimes/catalog)
Feature picker (searchable, powered by /api/v1/features/catalog)
mise tool picker (searchable, same /api/v1/runtimes/catalog payload)
A side-by-side preview of the generated devcontainer.json and mise.json blobs

Catalogs are refreshed on a 3-tier cache: in-memory → BoltDB → containers.dev scrape, TTL 24 h.

# Show current config
crewship crew config engineering --show

# Set devcontainer.json from file
crewship crew config engineering --devcontainer ./devcontainer.json

# Set mise.json from file
crewship crew config engineering --mise ./mise.json

# Change the base image
crewship crew config engineering --runtime-image debian:bookworm-slim

# Export full config (prints JSON)
crewship crew config engineering --export

# Clear all runtime config (back to default base, no features/mise)
crewship crew config engineering --clear

These flags are mutually exclusive — pick exactly one action per invocation.

PATCH /api/v1/crews/{crewId}
Content-Type: application/json

{
  "runtime_image": "debian:bookworm-slim",
  "devcontainer_config": "{\"image\":\"debian:bookworm-slim\",...}",
  "mise_config": "{\"tools\":{\"node\":\"22\"}}"
}

Server-side validation rejects blobs with whitespace/control chars in the image ref or non-allowlisted feature IDs. JSON config has per-field size limits enforced by the API:

devcontainer_config: 100 KB max
mise_config: 10 KiB for configuration; an optional native lock has its own 512 KiB decoded-file budget and 4 MiB JSON envelope limit

Configs larger than these limits are rejected at the API layer with HTTP 400. This prevents accidentally storing huge configs (often a sign of pasted-in log output or wrong field).

Triggering provisioning

Provisioning is asynchronous. The CLI command returns immediately after the backend enqueues a job; you then poll for status. Five triggers can put a crew into provisioning state:

Proactive auto-provision on config save — when a crew is created (POST /api/v1/crews, crewship apply, crewship crew create) or its devcontainer / mise / runtime-image config changes (PATCH /api/v1/crews/{crewId}), the handler immediately kicks off the build in the background (CrewHandler.maybeAutoProvision). This is the primary path: by the time anyone dispatches an issue the image is usually already cached, so operators never touch a “Build now” button. Editing the config invalidates the old cache selection and prepares an updated image. Managed Docker starts wait for current reservations and replace the container only after confirming it is idle. Unresolved occupancy leaves the update pending; an explicit stop/restart remains available after finishing existing work.
Explicit CLI/API trigger — crewship crew provision <slug> or POST /api/v1/crews/{crewId}/provision.
Crew rebuild — crewship crew rebuild <slug> bypasses the provisioned-image cache and installation-layer caches. It re-runs provisioning, so floating tool selectors can resolve newer versions; explicit versions in the configuration are retained. A rejected request (already provisioning or rate limited) leaves the current cache marker intact. This builds an image; it does not itself replace an already running container.
Auto-provision on first chat (PR #230) — when an operator opens a chat with an agent in a crew that has a config but no cached image, the chat handler calls ProvisioningHandler.EnqueueForCrew directly. The chat surface renders an inline CrewProvisioningCard with the live progress checklist; once provisioning succeeds, the agent’s first message starts on the freshly cached image.
Auto-provision on dispatch — when an issue, mission, or routine is dispatched to an agent in a crew that needs provisioning but has no usable cached image (never built, or the cache tag was pruned from the daemon), the dispatch path (AssignmentHandler.runAssignment → ProvisioningHandler.EnsureProvisioned) blocks until the image is built, then starts the agent on it. The same provision.* workspace events fire, so the top-right toolbar provisioning popover lights up (“preparing container”) with no extra wiring. If the build fails or times out, the run finishes with a clear “preparing the crew container failed: …” message instead of the cryptic exit 127 you’d get from launching the agent on a bare base image with no claude CLI. This guarantee means a crew is runnable the moment it’s created — operators don’t have to remember to crewship crew provision first.

The EnqueueForCrew API is idempotent: if a job is already running for the crew, the second call returns the existing job’s ID rather than starting a duplicate. Rate limiting is enforced — ErrRateLimited is returned as RFC 7807 Problem Details if a workspace is enqueueing more than the per-minute cap.

Trigger

crewship crew provision engineering
# ✓ Provisioning started for crew "engineering".

Under the hood: POST /api/v1/crews/{crewId}/provision → ProvisioningHandler.ProvisionTrigger → spawns a background goroutine that calls provisioner.Provision(ctx, baseImage, cfg, miseConfig).

Poll status

crewship crew provision status engineering
# Status        running        # pending | running | success | failed | idle
# Has Config    yes
# Cached Image  crewship-cache:a1b2c3d4e5f6
# Config Hash   a1b2c3d4e5f6…

The hash is a deterministic SHA-256 of (baseImage, devcontainer_config, mise_config) plus the generated-build fingerprint, the crew’s required adapter CLIs and the provisioner schema version — see configHash in internal/devcontainer/provisioner_cache.go. Any change to those invalidates the cache.

Re-build on demand

crewship crew rebuild engineering

Bypasses the provisioned-image and installation-layer caches and re-runs the build. The previously selected image stays in place until the new build succeeds — a failed rebuild leaves it selected. Use this when upstream features publish breaking updates under the same tag.

What `Provision` does, step by step

The entire pipeline is in internal/devcontainer/provisioner.go:Provision:

1. Compute config hash, check cache

hash := configHash(baseImage, cfg, miseConfig)
tag  := "crewship-cache:" + hash[:12]
if p.IsCached(ctx, hash) {
    return &ProvisionResult{CachedImage: tag}, nil
}

If the image tag already exists in the local registry, skip straight to the “already provisioned” return.

2. Skip if nothing to do

When the config has no features, no postCreateCommand, no containerEnv, and no mise tools, Provision returns CachedImage="" — the runtime launcher uses the bare RuntimeImage as-is.

3. Pull base image if missing

ensureImage runs ImageList then ImagePull if absent. The stream is drained to completion before proceeding — otherwise the next ContainerCreate fails with No such image.

4. Create a temporary container

Named crewship-provision-{hash[:8]}-{unixnano}, entrypoint ["sleep", "infinity"], runs as root so install.sh scripts can write everywhere.

5. Install devcontainer features

For each feature: {} entry:

Resolve feature ID → OCI artifact (ghcr.io/devcontainers/features/common-utils:2).
Fetch manifest, pull the single layer (media type application/vnd.devcontainers.layer.v1+tar, raw tar — not gzipped).
Write the tar stream into /tmp/devcontainer-features/{featureId}/ inside the container via CopyToContainer.
Exec install.sh with the feature options injected as env vars (e.g. USERNAME=agent USERUID=1001 /tmp/devcontainer-features/common-utils/install.sh).

Ordering honours installsAfter (topological sort on feature IDs) — see features.go:SortFeatures. Legacy wild-form [{id: string}] metadata is accepted alongside the spec-compliant []string.

6. Install mise tools (if declared)

Runs curl -fsSL https://mise.run | sh as the agent user, then mise install for each {tool: version} entry. Fails the provisioning job if any tool download 404s or the version is invalid.

Supply-chain risk: curl | sh from mise.run executes whatever that endpoint serves at provision time — no signature, checksum, or version pin. Provisioning also requires outbound HTTPS to mise.run from the build container. For high-security environments, vendor a known-good mise release into the base image (or a custom feature), pin by SHA256, and drop the mise section from devcontainer_config so this step is skipped.

7. Run postCreateCommand

Each command executes as UID 1001 (agent) via docker exec, with stdout/stderr streamed to the server log. Non-zero exit aborts the whole provision.

8. Clean up, commit, tag

Removes apt/pip/npm caches to shrink the committed image, then:

_, err := p.docker.ContainerCommit(ctx, containerID, container.CommitOptions{
    Reference: "crewship-cache:" + hash[:12],
})

The temp container is force-removed after commit (deferred).

Running agents on the cached image

Every path that starts a crew — chat, crewship issue start, a scheduled agent, a webhook, a routine’s agent or script step, the web terminal, the dashboard’s start button — resolves the crew’s runtime config through the same crew-start contract (internal/crewstart) and therefore boots the cached image. The terminal is worth calling out: it is the surface an operator opens specifically to inspect a crew’s environment, and it used to pass only the crew’s id and slug, so it started debian:bookworm-slim with none of the crew’s provisioned toolchain — gh, node and the agent CLI all missing — while crewship crew provision reported success (#1717). If you are debugging a crew through the terminal on an older build, check docker inspect <container> --format '{{.Config.Image}}' before trusting what the shell tells you about the toolchain. Underneath, container.EnsureCrewRuntime(ctx, team, ...) in internal/provider/docker/docker.go:

Resolve which image to boot:
- If team.CachedImage != "" → use the cached image.
- Else → use team.RuntimeImage or config default (debian:bookworm-slim).
Ensure Docker network crewship-agents exists (Internal: false — containers need outbound HTTPS for LLM providers).
ContainerCreate with:
- Entrypoint forced to /usr/local/bin/entrypoint.sh (bind-mounted).
- HostConfig:
  - CapDrop: ALL, CapAdd: NET_RAW, no-new-privileges, ReadonlyRootfs: true.
  - Memory / CPU / PidsLimit: 200.
  - ExtraHosts: ["host.docker.internal:host-gateway"] — lets the sidecar reach crewshipd on Linux too.
  - Mounts: /workspace, /output, /crew (rw binds), /secrets (in-memory tmpfs — 16 MiB, mode 0700, uid/gid 1001, never host-backed), /home/agent + /opt/crew-tools (named volumes), plus the two read-only bind mounts for the sidecar binary + entrypoint.
ContainerStart.

Sanity-check the sidecar bind mount for BYOI images:

if team.Image != "" && team.CachedImage == "" {
    // exec `ls /usr/local/bin/crewship-sidecar`; exit != 0 -> fail loudly
}

Catches Alpine/musl base images that silently can’t run the glibc-linked sidecar.

From then on, agent execution uses docker exec (never docker run).

Seed data: end-to-end demo

The crewship seed command seeds demo crews with sensible devcontainer + mise configs, then provisions all of them in parallel:

crewship seed                     # populate crews + credentials + agents
crewship seed --smoke-test        # also run `crewship run <slug> "hello"` on every agent
crewship seed --smoke-test --smoke-timeout 90   # per-agent timeout in seconds

Crew	Base image	Features	mise
Engineering	`mcr.microsoft.com/devcontainers/base:bookworm`	`common-utils`, `claude-code`, `github-cli`	`node:22`, `python:3.12`
Quality	same	`common-utils`, `claude-code`, `python:1`	—
DevOps	same	`common-utils`, `claude-code`, `docker-in-docker`, `kubectl-helm-minikube`	`terraform:1.9`
Research	same	`common-utils`, `claude-code`, `python:1`	`python:3.12`

The parallel provisioning has a 5-minute timeout per crew and a 3-second poll interval. Failed crews are logged as warnings but do not abort the rest — a partial demo is better than none.

The --smoke-test flag is the fastest way to prove the whole stack works after a fresh checkout: CLI → API → orchestrator → container → agent → LLM. If it passes, you can ship.

Container actuals

devcontainer.json declares what the container should look like. After agents have run a session — installing packages with apt-get, pip, or npm — the container’s actual state usually drifts from the declared intent. PR #231 closes that gap with the container.snapshot journal entry. After every successful agent exec, internal/containerstate.Snapshot runs four short probes inside the crew container:

Probe	What it captures
`dpkg-query -W -f='${Package} ${Version}\n'`	All apt-installed packages with versions
`pip freeze`	Python packages globally and in venvs on `$PATH`
`npm ls -g --json`	Globally installed Node packages
`cat /etc/os-release`	OS family, version, and `PRETTY_NAME`

Every probe is soft-fail: missing binaries (no pip in a Node-only image, no npm in a Python-only one) yield empty lists rather than errors. The snapshot is hashed (SHA-256 over the canonical-sorted package set + os details). The orchestrator emits a container.snapshot journal entry only when the hash changes — so quiet sessions that don’t mutate the container produce no churn at all. A typical session that adds one Python dependency emits exactly one container.snapshot entry (the new pip line), which then survives compaction the same way other observability entries do — see Crew Journal — container.snapshot for the payload schema. Operators can use these entries to diff intent vs reality:

“What did this agent install last week?” — query entry_type=container.snapshot on the crew, sort by ts.
“What’s drift between today’s container and the cached image?” — compare the latest snapshot against the manifest baked at provision time.

The probes always execute as the agent UID (1001), so they reflect what the agent could see — not what root could see. This is the right boundary for “what does the agent’s environment actually look like”.

Variable expansion in mounts and env

PR #225 extends the devcontainer expansion vocabulary the spec defines:

${devcontainerId} — resolves to a stable, Docker-volume-safe identifier derived from the crew ID (SHA-256 of the crew ID, first 16 hex chars). Useful for naming per-crew persistent volumes.
${VAR} / ${containerEnv:VAR} — resolved against the base image’s environment (everything in the image’s ENV directives), not the host’s. So referencing ${HOME} resolves to whatever the image set it to (e.g. /home/agent), not the operator’s home directory. Only the curly-brace form is expanded — bare $VAR is left untouched.

${devcontainerId} is expanded in mount source/target strings; ${VAR} / ${containerEnv:VAR} is expanded in containerEnv values (against the image’s default ENV). Unknown variables are left in place verbatim (e.g. ${TYPO} stays as the literal token) so an operator can spot and debug them in the rendered config — they are not treated as a hard provisioning error.

Runtime bind-mount semantics

crewship-sidecar + entrypoint.sh are bind-mounted read-only from the host into every crew container. Host-side edits do not take effect in already-running containers — the mount is snapshotted at docker create time.

To roll out a new sidecar or entrypoint across existing crews, rebuild the sidecar (make build:sidecar) and then remove each crew’s container: docker rm -f crewship-team-<slug>. The container is recreated transparently on the next agent message, picking up the updated bind mount.
Containers still serving an old sidecar are flagged on the next agent run as a sidecar.stale severity-error journal entry. Servers built with make build / make build:go carry the expected sidecar hash baked in at build time, so a deploy that ships a new server but forgets to rebuild/recopy crewship-sidecar is also flagged (plus a stale sidecar ARTIFACT detected server log) — that variant needs the sidecar binary redeployed first; recreating containers alone remounts the same old file.
postStartCommand runs on every start, so it is the right hook for operations that must reflect host state at the moment of launch (e.g. refreshing a secret via vault read). Do not try to bake those into postCreateCommand — they’d freeze at provisioning time.
Cached images (crewship-cache:{hash}) live on the Docker host indefinitely. Run crewship crew cache prune --older-than 30d as part of your maintenance rotation; cached images referenced by live crews are protected automatically.

Troubleshooting

docker build failed for crewship-feat:… — where is the reason?

It is in the error itself. A failed feature build appends a bounded tail of the daemon’s own output, so the message names the failing step rather than only its exit code:

Provision: building feature image: docker build failed for crewship-feat:3d097b41b405: exit status 1
docker build output (tail):
#5 0.688 E: Unable to locate package definitely-not-a-package
#5 ERROR: process "/bin/sh -c apt-get install -y …" did not complete successfully: exit code: 100
ERROR: failed to solve: process "/bin/sh -c apt-get install -y …" did not complete successfully: exit code: 100

The tail is the last part of the log, because that is where a BuildKit failure reports itself, and it is capped so a multi-megabyte build log never lands whole in an error string. Credential-shaped tokens are redacted on the way out — a RUN that echoes a build-arg or an API key shows [REDACTED].Need more than the tail? Two places have the rest:

crewship crew provision status <crew> — the durable provisioning.build_failed journal row keeps a larger window, and it survives after the live stream is gone.
Start the server with debug logging to stream every build line as it happens.

“the build produced no output” instead of a tail is its own finding: the docker CLI exited without printing anything, which almost always means it never reached a daemon. Check crewship doctor.

sidecar binary not found; run 'make build:sidecar'

autodetectSidecarPaths in internal/config/config.go failed. Run make build:sidecar on the host, or set CREWSHIP_SIDECAR_PATH=/absolute/path/to/crewship-sidecar and CREWSHIP_ENTRYPOINT_PATH=/absolute/path/to/entrypoint.sh. For tests that never launch containers, set CREWSHIP_SKIP_SIDECAR=1.

Nothing is wrong with your registry credentials — crewship-feat:* and crewship-cache:* are local-only tags that exist on exactly one daemon. This message means the create went looking for a locally-built image on a daemon that never built it, so it fell back to pulling.Historically that happened when the build (docker CLI, docker context) and the create (the provider’s probed socket) landed on different daemons — the default state on a machine where colima start or Rancher Desktop took the current context. The builder is now pinned to the provider’s endpoint, so the split cannot occur.If you still see it, the image really was removed from the provider’s daemon (a docker system prune on that daemon, most often). Re-provision:

crewship crew rebuild my-crew

Check which daemon Crewship is on with crewship doctor or GET /api/v1/system/runtime, and list the tags there with DOCKER_HOST=<that socket> docker images | grep crewship-.

sidecar bind mount sanity check failed (exit 126)

The user-provided base image is musl-based (Alpine) or missing glibc. Switch to a glibc image:

crewship crew config my-crew --runtime-image debian:bookworm-slim
crewship crew rebuild my-crew

feature 'xyz' install.sh exits non-zero

Most common cause: base image lacks prerequisites (curl, git, ca-certificates). debian:bookworm-slim is intentionally minimal. Either:

Switch to mcr.microsoft.com/devcontainers/base:bookworm (pre-installed), or
Declare ghcr.io/devcontainers/features/common-utils:2 as the first feature — it installs prerequisites.

base/runtime image freshness unknown; keeping the local copy without pulling

Before pulling, Crewship asks the registry for the current manifest digest (a HEAD) and compares it against the local image’s RepoDigests. If that lookup fails, the pull is skipped and the local copy is used as-is — the right call when you’re offline, but it means a permanently failing lookup pins you to a stale image indefinitely. That case now logs at WARN:

WARN runtime image freshness unknown; keeping the local copy without pulling — it may be stale.
     Check registry reachability and your Docker credential helper (~/.docker/config.json credsStore)
     ref=debian:bookworm-slim

Two usual causes:

No route to the registry (air-gapped host, egress firewall). Pre-pull images on the host and ignore the warning.
A broken or wedged Docker credential helper. ~/.docker/config.json names a credsStore, and Docker execs docker-credential-<store> for every registry. If that binary is missing, hangs, or prompts for an unlocked keychain, the digest lookup times out (5s). Verify with:
echo "ghcr.io" | docker-credential-$(jq -r .credsStore ~/.docker/config.json) get
Digest lookups for registries that need no credentials — loopback/localhost registries — skip the helper entirely, and a helper that errors falls back to anonymous access rather than giving up, so public images keep resolving.

cache image never refreshes

configHash is deterministic over (baseImage, devcontainer_config, mise_config) plus the generated-build fingerprint, required adapter CLIs and the provisioner schema version. Upstream feature releases that re-use the same tag don’t change the hash. Force a rebuild:

crewship crew rebuild my-crew

tmux: not found (noisy log on startup)

Fixed in commit 44da863: setupTmuxExec now pre-checks command -v tmux and falls back to stdbuf -oL <cmd> when absent. debian:bookworm-slim doesn’t ship tmux — this is expected.

claude: not found inside agent container

The Claude Code CLI is installed via the ghcr.io/devcontainers-extra/features/claude-code:2 feature (replaces our former Go EnsureClaudeCode helper). Make sure it’s in the crew’s devcontainer_config.features. Seed crews include it automatically.

Migrating from the legacy `agent-runtime` image

If you have a long-running deployment that still references ghcr.io/crewship-ai/agent-runtime:latest:

Update Crewship to the current release

Post-dd86356.

Build the sidecar on the host

make build:sidecar

Reconfigure each existing crew

crewship crew config <slug> --runtime-image debian:bookworm-slim
crewship crew config <slug> --devcontainer ./devcontainer.json   # with common-utils + claude-code
crewship crew rebuild <slug>

Restart crewshipd

The fail-fast check will confirm your sidecar is discoverable.

The legacy image is no longer published by CI (agent-runtime.yml workflow deleted). Pulling its old tag still works but sits frozen; nothing in Crewship itself assumes it anymore.

Reference

Configuration → Devcontainers — config fields + environment variables reference.
Configuration → Environment Variables — CREWSHIP_SIDECAR_PATH, CREWSHIP_ENTRYPOINT_PATH, CREWSHIP_SKIP_SIDECAR.
Security → Container Isolation — UID boundaries, capability drops, network policy.
Code of record: internal/devcontainer/, internal/provider/docker/docker.go, cmd/crewship/seeddata/crews.go, cmd/crewship/cmd_crew_config.go, cmd/crewship/cmd_crew_provision.go.

Built environment history

crewship crew provision revisions <crew> lists the latest 100 immutable build records for a crew. The API is GET /api/v1/crews/{crewId}/provision/revisions. Each record identifies the definition hash, build hash, immutable image ID, observed CLI inventory and build time. Records are scoped to the crew’s workspace; raw configuration and environment variable values are not copied into history. Reusing the same image for the same definition reuses its record. Offline qualification is included in that immutable build evidence; it is not a current provider-authentication claim. By default, offline AI CLI checks record evidence without preventing publication. To require them for a crew, set spec.mise.ai_cli_check: required in its Crew manifest, or "ai_cli_check":"required" alongside tools in the managed mise_config JSON. record explicitly selects the compatibility default; other values are rejected. This is Crewship metadata and is not emitted into native mise TOML. The crew creation wizard preserves this policy when importing Crew YAML, even when the mise section has no explicit tools. A nonempty typed ai_cli_check overrides the value in spec.mise.raw; unsupported typed values are rejected. A required check accepts only successful evidence bound to the built immutable image, with a passing startup/version probe for every recorded and required agent CLI. Missing inventory, an unavailable runtime, mismatched evidence or failed probes fail the provisioning job before a new revision/current image is published. The previous image and requirements stay selected. Native providers without the necessary inspection and offline sandbox capability cannot satisfy this opt-in requirement. These checks do not prove account/model compatibility, real provider authentication or safe automatic activation. Workspace and crew backups include these records; restoring them preserves provenance and does not restore the referenced image or activate an instance. A cache hit whose feature requirements cannot be resolved fails provisioning and leaves the previous selection intact. A successful empty build clears the previous artifact’s runtime requirements and feature evidence. A completed build verifies that the crew definition and required agent adapters still match the inputs it built. If they changed, Crewship refuses to publish the stale result and retries the current definition within the same job, retaining pending messages. Retries share the original 30-minute budget and stop after eight superseded attempts; ongoing edits or a deleted crew produce a terminal failure without replacing the previous selection. The check and publication are one database transaction. This history is provenance metadata. It neither preserves Docker images from pruning nor changes a running container, and it does not provide rollback or RAM snapshots. Builds without recorded immutable image identity do not invent a revision. These records will support a later explicit retention and safe activation policy.

Immutable Docker build references

Docker provisioning records a full local image ID in cached_image. A shared crewship-cache:* tag is only a cache lookup alias; moving it during a later build does not change what a previously provisioned crew selected. A failed explicit rebuild leaves the previous reference and requirements intact. Builds still in progress do not replace the running container. The build cache key changed once for this transition: new builds use new aliases so they do not move tags retained by older crew records. Existing records are not rewritten. Native Apple builds still return tags because that provider does not yet expose equivalent image inspection through this interface. The provisioning API’s cache inventory, deletion and orphan sweeper recognize crew references by image ID. The separate opt-in idle cache eviction policy also keeps images selected by live crews in this installation, including legacy tag aliases. Revision history does not retain image bytes or guarantee rollback. External docker image prune, forced admin deletion and disk loss also remain outside this protection. A bare provider start requesting a different image returns image update pending while the current container is running, paused or its state is unknown. Managed starts reserve the runtime through preparation and execution; an image change waits for existing reservations, then verifies installation ownership and controller/runtime occupancy before stopping the old instance. A freshly inspected stopped container can be removed without force and recreated. Selecting the same immutable artifact by ID does not require a restart even if the container was created using a tag. A runtime confirmed to be in restart backoff retains the separate crash-recovery path. The orphan sweeper also recognizes published image IDs recorded in revision history after a rebuild removes their last tag. With cache auto-deletion enabled, the existing age floor and live-crew reference checks apply to those artifacts too; removal remains non-force. Unrecognized images without tags are left alone. An unpublished build, or an artifact whose history was removed with its crew, cannot be identified this way. Revision history is not a promise to retain an old image for rollback. Legacy agent runs inspect their actual container at admission and retain its immutable source image ID as runtime_image_id in run state and exec.command events. The lookup has a two-second timeout; unavailable evidence is omitted, never inferred from the crew’s currently selected image. This identifies the container’s source, not subsequent filesystem changes, the executed CLI’s version, or a unique environment definition revision. Restricted attempts have their own frozen-input evidence; this field does not change their contract. Automatic activation requires no local holds, local or unresolved recovered invocations, external exposures, or additional workload processes. A bounded kernel process census recognizes the standard sleep/init entrypoint and the trusted sidecar UID. Missing state, incomplete observations and custom resident process trees keep the update pending. Idle expiry uses the same reservation exclusion. Terminal sessions retain the existing runtime rather than selecting a pending image, and prevent its replacement while connected. This is controller-local coordination for managed runtime users, not a guarantee against an operator concurrently using Docker directly or replacing the daemon. Providers without managed admission do not gain automatic activation. Explicit operator restart remains disruptive; finish existing work before using it to resolve a pending update. No scheduled updater or crash-safe output replay is implied by this activation path.

Supplying a native mise lock

The managed build can consume a complete native mise lock bundle. Generate mise.lock with mise for the container’s target platform, then run:

crewship crew provision lock-pack /path/to/mise-config-directory > lock-bundle.json

This local command reads only mise.lock and .mise/locks/, rejects symlinks, limits file sizes, and does not contact Crewship or change the environment. Keep auxiliary files: npm-backed tools may use an aube lock beside mise.lock. The bundle supports up to 128 UTF-8 files, 128 KiB per file and 512 KiB total. Put the resulting JSON object in spec.mise.lock in a kind: Crew manifest, alongside spec.mise.tools. JSON configuration stored in mise_config accepts that same lock object. Manifest export/import and the visual tools editor preserve it. Raw TOML input still supports the existing [tools]/[env] subset; the lock bundle is a Crewship JSON/YAML field, not a new TOML section. With a bundle, builds install using mise install --locked --force --yes. An absent version/platform, invalid lock or checksum mismatch fails the build; it does not fall back to a floating install. Force reinstalls tools so an already-installed version in the base image cannot bypass this check. Added AI adapters still need matching entries in the supplied lock. Changing a selector in the visual editor preserves the lock and may therefore require regenerating it before the next build. Remove the lock explicitly to return to ordinary unlocked installation. Mise owns resolution and backend semantics. A native lock without an upstream checksum does not gain one from Crewship, and support varies by backend. The bundle does not lock the base image, OS packages, features, build hooks or the mise executable itself. It is dependency input, separate from observed CLI versions and from qualification of real model/auth combinations. This slice adds no scheduled updates; runtime activation is governed separately above.

Offline CLI qualification

On Docker, provisioning also checks the recorded CLI binaries in a disposable instance pinned to the built image ID. The instance runs as UID 1001 with no external network, no persistent mounts, a read-only root filesystem and bounded CPU, memory, process count, output and execution time. Its temporary home is removed after the check. Crewship supplies no provider credentials. The qualification requests a 90-second lifetime; the initial backend accepts at most two minutes and uses a separate UID 1002 for its lifetime keeper. This temporary profile disables inherited image healthchecks and is not a durable agent-run backend. toolchain.built.qualification reports the image ID, per-tool result and overall status. passed means that the recorded binaries started and reported the same versions in this offline profile. It does not verify provider login, model compatibility, streaming, or isolation of ordinary crew runs. A failed or unavailable check is reported without replacing an active environment or silently relaxing the qualification profile. Older images and unsupported backends can report unavailable. The initial runtime port refuses persistent mounts and RAM suspension explicitly.

Resolving or updating a lock with Crewship

crewship crew provision lock-resolve mise.json --image sha256:<local-image-id> resolves a mise JSON/TOML tool configuration in a disposable Docker builder. Supply a trusted, already-present Linux image containing /usr/local/bin/mise, /usr/bin/env, /bin/sleep and /bin/tar. No image is pulled automatically. The resolver selects linux-x64 or linux-arm64 from that image’s architecture; it does not infer the target from the CLI host’s operating system. The JSON result contains lock, resolver_image_id, platform, mise_version and tools with native previous/resolved version lists and version_changed. selectors_sha256 binds the requested tool selectors; previous_lock_sha256 (when supplied) and lock_sha256 cover every file in the complete bundle. lock_changed includes source, checksum, auxiliary-file and formatting changes, so it can be true even when no tool version changes. These hashes identify the proposal’s inputs and output; they are not signed approval or authorization. Apply its lock through spec.mise.lock and the ordinary build path. For an update, include the previous lock alongside tools in the input JSON and pass --bump. Mise updates within each selector; an exact version remains exact. The result is a proposal, with no running-crew or server changes. This is trusted build work with Docker bridge networking for registries. It uses a read-only root, no persistent or host mounts, UID 1001 for resolution, dropped capabilities, bounded memory/CPU/processes, a private 256 MiB temporary filesystem and a five-minute operation deadline. A separate UID-1002 keeper exits after six minutes if the client disappears; stopped-container cleanup still depends on the client or provisioning cleanup. The image, its tool backends and the Docker host remain trusted. This is not the restricted agent network policy or hostile-image execution. Configuration environment values are excluded; private registry credentials and host mise configuration are not inherited. Input is limited to 4 MiB; native bundles retain the limits above, and resolver diagnostics are bounded. Resolution does not install the resulting toolchain, prove a real provider login, schedule an update or activate a revision. The required check also compares explicit exact mise CLI selectors with the observed version. An overlapping devcontainer feature does not hide an explicit mise pin. Floating selectors remain the responsibility of native locked install; this check does not implement another dependency resolver.

Python tools need a prepared resolver and builder

The resolver image must contain the native backend’s prerequisites as well as mise. For Python package graphs, prepare a compatible Python interpreter and uv in that trusted image; the build image must also provide the interpreter mise selects. Crewship’s resolver does not install these prerequisites or silently relax a locked installation. See the upstream PyPI backend requirements. In a live Ansible check, an image without uv produced only a top-level version record; the subsequent mise install --locked correctly rejected its missing Python graph. Adding uv without a sufficiently recent interpreter also failed. Treat resolution as a proposal, then verify installation and the actual tool before selecting the resulting environment. The Ansible distribution version and the ansible-core version printed by ansible --version are different identities. An explicit --bump can refresh dependency metadata even when an exact package version stays unchanged. Review lock_changed and the complete bundle, not only the top-level version diff. A normal resolution without --bump reuses the existing native lock; no update scheduler is introduced here.

Applying a reviewed native lock proposal

crewship crew provision lock-apply mise.json proposal.json prints updated mise JSON only if the current selectors and previous complete lock match the plan, and the proposed lock still matches its digest. Redirect to a new output file, then review and apply that configuration through your ordinary manifest or crew API workflow. The command is local and offline; it does not install software, start a build or replace a running environment. Environment and policy fields come from the current configuration, not from the resolver. Unknown JSON fields are preserved. A changed selector or old lock requires a new plan. Hashes prevent accidentally applying a stale or corrupted proposal; they do not authenticate its author or independently prove native resolution. The normal locked build remains responsible for checking that lock against the selectors. Provisioning can also require the separate image-bound offline AI CLI checks described above.

Viewing built CLI versions

Crew settings → Runtime shows requested AI CLI selectors alongside versions recorded in the selected build. Startup-check status belongs to that same image; older or unavailable evidence remains unknown. These observations do not claim that an already-running environment has switched image or that its writable files are unchanged. Provider sign-in is a separate check. The panel refreshes after this crew’s build completes or fails and after a realtime reconnect. Use its refresh button if live updates are unavailable. Viewers can inspect versions without permission to modify the environment.

Preserving configuration the visual editor cannot represent

The visual tool picker edits JSON with string tool selectors. Existing TOML, structured tool values, or malformed input remain unchanged when you edit another runtime setting. The picker shows the preserved source instead of offering controls that would discard it. Edit that configuration through your Crew manifest or CLI; the raw runtime editor keeps unsupported source read-only. Supported JSON edits preserve other fields, including environment, lock and CLI-check policy. A rejected raw edit applies neither configuration document.

Get Started

Guides

Security

Configuration

Manifest Reference

Devcontainers & Runtime Images

Devcontainers & Runtime Images

What you get

Bring Your Own Image (BYOI)

Community Features

mise Runtime Tools

Per-Crew Cached Image

Mental model

Phase 1: Dockerfile + BuildKit provisioning

Prerequisites

Configuration shape

Supported `devcontainer.json` fields

mise config

The agent’s PATH

Adapter CLIs are installed and verified

Managing a crew’s runtime config

Triggering provisioning

What `Provision` does, step by step

Running agents on the cached image

Seed data: end-to-end demo

Container actuals

Variable expansion in mounts and env

Runtime bind-mount semantics

Troubleshooting

Migrating from the legacy `agent-runtime` image

Reference

Built environment history

Immutable Docker build references

Supplying a native mise lock

Offline CLI qualification

Resolving or updating a lock with Crewship

Python tools need a prepared resolver and builder

Applying a reviewed native lock proposal

Viewing built CLI versions

Preserving configuration the visual editor cannot represent

​Devcontainers & Runtime Images

​What you get

Bring Your Own Image (BYOI)

Community Features

mise Runtime Tools

Per-Crew Cached Image

​Mental model

​Phase 1: Dockerfile + BuildKit provisioning

​Prerequisites

​Configuration shape

​Supported devcontainer.json fields

​mise config

​The agent’s PATH

​Adapter CLIs are installed and verified

​Managing a crew’s runtime config

​Triggering provisioning

​What Provision does, step by step

​Running agents on the cached image

​Seed data: end-to-end demo

​Container actuals

​Variable expansion in mounts and env

​Runtime bind-mount semantics

​Troubleshooting

​Migrating from the legacy agent-runtime image

​Reference

​Built environment history

​Immutable Docker build references

​Supplying a native mise lock

​Offline CLI qualification

​Resolving or updating a lock with Crewship

​Python tools need a prepared resolver and builder

​Applying a reviewed native lock proposal

​Viewing built CLI versions

​Preserving configuration the visual editor cannot represent

Devcontainers & Runtime Images

What you get

Mental model

Phase 1: Dockerfile + BuildKit provisioning

Prerequisites

Configuration shape

Supported `devcontainer.json` fields

mise config

The agent’s PATH

Adapter CLIs are installed and verified

Managing a crew’s runtime config

Triggering provisioning

What `Provision` does, step by step

Running agents on the cached image

Seed data: end-to-end demo

Container actuals

Variable expansion in mounts and env

Runtime bind-mount semantics

Troubleshooting

Migrating from the legacy `agent-runtime` image

Reference

Built environment history

Immutable Docker build references

Supplying a native mise lock

Offline CLI qualification

Resolving or updating a lock with Crewship

Python tools need a prepared resolver and builder

Applying a reviewed native lock proposal

Viewing built CLI versions

Preserving configuration the visual editor cannot represent