​

Container Isolation

​

Isolation Layers

+-----------------------------------------------------------+
| Layer 5: Keeper (AI-powered credential gating)            |
+-----------------------------------------------------------+
| Layer 4: Network Policy (free/restricted domain filtering)|
+-----------------------------------------------------------+
| Layer 3: Sidecar Proxy (credential injection, scrubbing)  |
+-----------------------------------------------------------+
| Layer 2: UID Boundary (agent=1001, sidecar=1002)          |
+-----------------------------------------------------------+
| Layer 1: Container Runtime (Docker, gVisor, Kata, Sysbox) |
+-----------------------------------------------------------+

​

UID Security Boundary

​

What Agents Cannot Do

• Agents cannot read the sidecar’s in-memory credential store
• Agents cannot access the IPC token (used for crewshipd authentication)
• Agents cannot modify network policy configuration
• Agents cannot tamper with outbound request headers (the sidecar handles this)
• Agents cannot set their own container ID in Keeper execute requests

​

The server’s own Docker access

​

Privileged crews and credentials (fail-closed)

• UI: Settings → General → Privileged credentials → “Load credentials into privileged crews.” (It is workspace-wide, not per-crew — a crew’s own Settings tab configures that crew, this switch governs the whole workspace.)
• API: PATCH /api/v1/workspaces/{id} with {"allow_privileged_credentials": true} (read it back on GET /api/v1/workspaces/{id}).
• CLI: crewship workspace update --allow-privileged-credentials.

​

Container Runtime Security

container:
  default_runtime: "runsc"  # or "runc", "kata-runtime", "sysbox-runc"

​

Container Creation Security

• Cap-drop ALL: All Linux capabilities are dropped (CapDrop: ["ALL"])
• No default cap-add: By default no capabilities are re-added. NET_RAW was removed from the default set (it enables AF_PACKET/DNS-tunneling exfiltration). Devcontainer features may opt into an explicit allowlist (e.g. NET_BIND_SERVICE) via team.CapAdd.
• SecurityOpt no-new-privileges: Blocks execution of setuid binaries inside the container
• SUID stripped from the base image: The crewship-sandbox image strips the SUID bit from every setuid-root binary (su, mount, umount, passwd, chsh, newgrp, ssh-keysign, …) at build time. Only sudo is kept — it is the elevation path for the NOPASSWD firewall-init rule and sudo apt-get package-install flows. This is defense-in-depth behind no-new-privileges (which already neuters setuid): it shrinks the surface that a single regression dropping NoNewPrivs or granting a capability could re-arm. A build-time assertion fails the image build if any SUID binary other than /usr/bin/sudo reappears after a base-image bump.
• Non-root execution: Agent processes run as UID 1001; exec’d commands run as the container’s resolved configured user and are refused outright if that user would be root or cannot be determined
• Read-only filesystem: Root filesystem is read-only (ReadonlyRootfs: true), relaxed only for privileged crews (DinD)
• tmpfs /tmp (500M): Writable temp directory (rw,noexec,nosuid,size=500m), not persistent across restarts. rw still permits staging a file here, but noexec blocks executing it directly and nosuid strips any setuid/setgid bit; scripts still run via an explicit interpreter (sh, bash, …), which reads the file rather than exec’ing it.
• noexec on agent-writable persistent volumes: /workspace, /output and /crew are agent-writable and host-persistent — /crew in particular is a host directory that survives container removal and is shared across every agent in the crew. They are mounted noexec,nosuid, so a compromised agent can still write data there but cannot execve() a payload staged on them. This closes the “durable, cross-agent code-execution foothold” vector (a payload dropped once under /crew/shared outliving container rebuilds). Docker’s bind-mount API has no noexec flag, so these ride a bind-backed local volume (type=none,device=<host path>,o=bind,noexec,nosuid) — the mount still binds the exact host directory (data persists, still crew-shared), just with execution denied. The exceptions are deliberate: /opt/crew-tools stays executable because agent tool binaries run from there, and the sidecar/entrypoint binaries are mounted read-only. Scripts under these volumes still run via an explicit interpreter (sh <path>), which reads the file rather than exec’ing it, so the runtime is unaffected. Existing containers created before this policy are recreated on next provision so noexec takes effect. /home/agent is a deliberate exemption — see The /home/agent exemption below.
• PID limit: 200: Prevents fork bomb attacks (PidsLimit: 200)
• Memory limits: Configurable per-crew (default_memory_mb, default 8192 MB / 8 GiB)
• CPU limits: Configurable per-crew (default_cpus, default 2.0)
• Shared network, restricted egress: every crew’s container joins one Docker bridge network (crewship-agents by default) — crews are not network-isolated from each other on that bridge. Outbound access is controlled per crew by egress policy enforced by the sidecar proxy (127.0.0.1:9119, via HTTP_PROXY/HTTPS_PROXY, plus NODE_USE_ENV_PROXY=1 so Node’s built-in fetch honours them on Node 22.21+ and 24+) — it filters only traffic actually routed through it. A raw socket, or a tool pointed past the proxy (curl --noproxy '*'), reaches the bridge directly: the shared bridge itself carries no network-layer isolation to fall back on. A crew listed in container.crew_network_crews gets its own bridge instead (IPv4 only, a /27 from container.crew_network_pool): containers of other crews, on their own networks or on the shared one, cannot reach it, and its services: aliases (postgres, redis) resolve to its own containers. Crews talk to each other through the Crewship API, not over the network. A crew network is internal whenever the instance network is. Port-expose only publishes containers of the requesting crew. Crew networks need crewshipd running on the Docker host itself, with a local daemon: a containerised crewshipd (the production compose file) is not attached to crew networks, so a listed crew there fails to start with crew networks unsupported here instead of running unable to reach Crewship. If the instance network does not exist, a crew network is refused rather than created with weaker isolation.
• ExtraHosts: host.docker.internal:host-gateway enables container-to-host communication
• Named volumes: crewship-home-{slug}-{crewID} and crewship-tools-{slug}-{crewID} for persistent home and tools directories (namespaced by crew ID to prevent cross-tenant collisions)

​

The /home/agent exemption

​

Runtime Escape Hatches

​

How enforcement is wired

1. Save (ValidateSecurity) — privileged requires allow_privileged_credentials (403); capAdd is bounded to NET_BIND_SERVICE (400); every mounts[].source must pass IsAllowedMountSource (400). The keys are persisted in the canonical config (they used to be silently dropped on the auto-inject re-marshal).
2. Runtime read (ParseConfigSecurity) — at container start the resolver folds the validated top-level keys into the effective requirements the provider maps onto the Docker HostConfig (internal/chatbridge/resolver.go → internal/provider/docker/docker_container.go), re-filtering caps/mounts as defense in depth. The #1032 credential-fail-closed gate reads the same source, so a UI-privileged crew is correctly treated as privileged when deciding whether to load vault credentials.

​

Seeing the posture

• Crew canvas header — a red Privileged · isolation reduced badge next to the crew name, on the first screen anyone opens.
• Security tab — an Isolation reduced badge plus a dot on the tab itself.
• CLI — crewship crew config <crew> --show prints the stored devcontainer_config, privileged included.

​

Setting them from the CLI

​

CLI parity

# Toggle privileged (needs the workspace allow_privileged_credentials flag, else 403)
crewship crew config my-crew --privileged

# Grant a capability (only NET_BIND_SERVICE is accepted; anything else → 400)
crewship crew config my-crew --cap-add NET_BIND_SERVICE

​

Network Policy

# No explicit network policy = free mode

{
  "mode": "restricted",
  "allowed_domains": ["github.com", "*.github.com", "registry.npmjs.org"]
}

crewship crew update my-crew --network-mode restricted --allow-package-registries

​

Private / LAN endpoints

1. Per crew — allow_private_endpoints. Set it from the crew’s Network policy panel (admin-only toggle) or with crewship crew update <crew> --allow-private-endpoints.
2. Per instance — the operator ceiling CREWSHIP_ALLOW_PRIVATE_ENDPOINTS.

​

One allowlist, every egress path

​

Network-layer egress fence (pilot)

• The container never holds NET_ADMIN. A one-shot helper container joins the crew’s network namespace with only NET_ADMIN, runs the bind-mounted crewship-sidecar --fence-apply, and exits. Nothing inside the crew container can remove the rules.
• Every start re-installs it. A container restart recreates the network namespace and drops the fence, so the runtime installs it after each start it performs and checks the container’s start time before reusing a running container.
• No command runs in an unfenced start. The container’s entrypoint runs nothing of the agent’s; agent code arrives only through an exec. Every exec into a fenced container, including the web terminal, the crew’s start-up hooks, backup flush hooks, restore scripts and memory imports, first confirms the fence for the container’s current start. (Image builds exec only into temporary build containers, never into a crew’s.) After a restart outside Crewship (a daemon restart, an operator, a crashed process) the fence is installed again before the exec runs; if the server has not fenced the crew before (for example right after a server restart) the exec is refused until the next crew start. A container the server has fenced stays fenced by its id even if its crew labels are missing or stale. The start is checked again just before the exec runs and once it runs; a restart in between refuses the exec, or stops the crew if it already started — also when the caller gave up in between.
• Connections do not outlive it. Established traffic is accepted only in the reply direction, for connections opened to the container. A connection the agent opened itself, including one opened before the fence went in, stops delivering data once it is installed.
• Failure is closed. A listed crew whose fence cannot be installed is stopped, not run unfenced.
• The host is untouched. The rules live only in the crew’s namespace; the host firewall and Docker’s own rules do not change.

• It applies to restricted crews on the runc runtime. A listed crew that is privileged, free, or on another runtime is refused: privileged mode removes the UID boundary the fence rests on, and gVisor and Kata carry traffic outside the namespace’s netfilter hooks.
• Tools that resolve hostnames themselves before using the proxy fail, because the agent cannot query DNS. Tools that honour HTTP_PROXY/HTTPS_PROXY (curl, git, npm, pip, the agent CLIs) let the sidecar resolve and keep working.
• Turning the pilot off for a crew (removing it from the list) takes effect when its container next starts; until then the rules stay in its namespace. Stop and start the crew (crewship crew stop <crew>, then crewship crew start <crew>) to apply the change at once.
• The fence controls traffic leaving the container, and replies to connections opened into it. Inbound reachability from other crews on the shared network is tracked separately in #2240.

​

Supported and refused combinations

​

Reaching the crew’s own services under the fence

• Fixed addresses. On a crew’s own /27 each service has a fixed address (.4 onward, by the sorted service names); every other container on that network gets an address from the upper half. A service address is never handed to another container, and a recreated service comes back at the same one.
• Exact endpoints. The fence accepts packets to each service’s address on its declared ports and the ports its image exposes, by protocol. Not the subnet, not other ports, not the host.
• Names without DNS. The agent cannot query DNS under the fence. Service names are written into the runtime container’s /etc/hosts when it is created, so postgres:5432 or redis:6379 resolve. Adding or renaming a service therefore takes effect when the crew’s container is next recreated (after a stop; a running container is not recreated under the agent).
• Kept current. After the crew’s services are ensured, the fence is re-checked and re-installed in one atomic swap if their endpoints changed (for example a newly exposed port). This also happens when ensuring the services fails part-way, so the fence always matches the declared services and never keeps the address a moved service left.
• Address changes. Because addresses follow the sorted names, declaring a service can shift the address of an existing one. The shifted service is detached and attached again at its new address; the old address closes in the fence. Until the crew’s container is recreated, its /etc/hosts still names the old address, where the port is closed: the service is unreachable by name, never reachable through another service.
• Failures close the path. A service that is down, cannot start, or whose move between networks fails is unreachable; the crew stays fenced and gains no other destination.

​

What the fence does not protect: credentials inside the container

​

Operating the pilot

• Turn on: add the crew’s slug or id to container.egress_fence_crews (or CREWSHIP_EGRESS_FENCE_CREWS) and restart the server. The fence goes in the next time the crew runs: on a container start, or on the first exec into an already running container.
• Changing the crew: a re-install after an outside restart uses the crew settings of the last install. Making a fenced crew privileged, free or giving it services takes effect when its container is next recreated.
• Check: the server log records egress fence installed with the crew id and the install time for every start. Blocked direct connections fail at once (No route to host, Operation not permitted, or a name-resolution failure); traffic through the proxy is unchanged and appears in the crew’s journal as network.egress.
• When a crew does not start: the log line crew requires the egress fence and it is not in place names the cause (an unsupported combination from the table above, or the install failing). An exec refused with egress fence not in place for this container start means the container was restarted outside Crewship and has not been fenced again yet; the next exec into a crew the server has fenced before re-installs it, and after a server restart the next crew start does.
• Turn off: remove the crew from the list, restart the server, then stop and start the crew. The rules live in the container’s network namespace and leave with it; removing the crew from the list alone does not remove them from a running container.

​

Domain Allowlist Implementation

• Domains are stored lowercase for case-insensitive matching
• Port numbers are stripped before comparison
• IPv6 bracket notation is handled correctly
• *.example.com entries are stored as a subdomain-suffix matcher (matches any subdomain, not the apex); everything else is an exact host match
• Domains can be added at runtime via allowlist.Add(domain) (exact or wildcard)

​

Proxy Security

​

Request Body Limits

const maxRequestBodyBytes = 10 * 1024 * 1024 // 10 MB

​

Hop-by-Hop Header Stripping

​

HTTPS CONNECT Tunnels

1. Checks the domain allowlist (in restricted mode)
2. Establishes a TCP tunnel
3. Does not inject credentials (the tunnel is opaque)

• Plain HTTP proxy requests (agent sets HTTP_PROXY)
• Reverse proxy mode — Claude Code via ANTHROPIC_BASE_URL=http://127.0.0.1:9119 (→ api.anthropic.com) and, since #1030, Codex via OPENAI_BASE_URL=http://127.0.0.1:9119/openai/v1 (→ api.openai.com) and Gemini via GOOGLE_GEMINI_BASE_URL=http://127.0.0.1:9119/gemini (→ generativelanguage.googleapis.com); the /openai / /gemini routing prefixes are stripped before forwarding. For all three, the real key stays in the sidecar CredStore and only a dummy placeholder reaches the agent env.

​

Provider Detection

func providerForHost(host string) ProviderType {
    switch host {
    case "api.anthropic.com":    return ProviderAnthropic
    case "api.openai.com":       return ProviderOpenAI
    case "generativelanguage.googleapis.com": return ProviderGoogle
    default:                     return ""  // no injection
    }
}

​

Credential Injection by Provider

​

Keeper Execute Security

1. Sidecar validation:
   • Intent and command length limits (4096 chars)
   • Null byte rejection
   • Shell metacharacter blocking outside single quotes (;, |, >, &, <, `, $(, newlines)
   • Content inside single quotes is exempt
   • Container ID and requesting agent always from IPC config and the agent’s bearer token (agent cannot override)
2. crewshipd validation:
   • A stricter second denylist (defense in depth): the operators above plus ${, <(, <<, ANSI-C $'…', every control character including U+2028/U+2029, interpreter re-invocation (sh -c, python -c, node -e, …) and awk/sed, with real quote-state tracking — see Shell injection protection
   • The credential must be ACTIVE, granted to the requesting agent and inside its lease, or the request is 404 before the judge runs
   • The Keeper judge evaluates the command; the L1 fast path never applies to execute
3. Execution:
   • Credential injected as environment variable for the command only, run as the container’s resolved non-root user (refused if that user is empty, undeterminable or root)
   • Output scrubbed of the credential value and a fixed set of its encodings (base64 variants, base32, URL-escaped, hex, reversed) before returning to agent — a denylist of transforms, not a boundary

​

Output Scrubbing

• Credentials appearing in chat messages
• Credentials being logged to the progress stream
• Credentials in Keeper execute output being returned to the agent

​

Container Auto-Detection