Deploying

Zimmer deploys with Kamal onto a single DigitalOcean droplet, reachable only over Tailscale. Terraform bootstraps the box (Docker, Tailscale, Caddy, the deploy key); Kamal owns the app stack — a web role and a worker role, with durable named volumes. There is no Kubernetes, no load balancer, and no HA. TLS is optional and off by default — setting var.domain adds a tailnet-only HTTPS front door (see below). An optional Cloudflare edge can serve that domain to the internet through an outbound-only tunnel, still with no public port open.

These docs cover the staging deployment, which is the one this repo operates. A production deployment is self-hosted and lives in your own private infrastructure — the config/deploy.yml / config/deploy.production.yml and .kamal/secrets.production here are public-safe templates, but the production environment (its DNS, its database, its secrets, and the workflow that drives it) is out of scope for these docs.

The topology

By default there is no TLS. The web container serves plain HTTP on :80 behind kamal-proxy, and production.rb sets assume_ssl and force_ssl, which works only because assume_ssl makes Rails pretend the request arrived over TLS. The actual encryption is WireGuard, via Tailscale. A future public ingress would break this subtly and badly.

Setting var.domain adds a real HTTPS front door (see below), still tailnet-only, which makes assume_ssl true in reality.

Custom-domain HTTPS over the tailnet

Plain HTTP with assume_ssl is a known sharp edge: because Rails computes https:// origins that never match the browser’s http://, every CSRF-protected form POST 422s and every ActionCable upgrade is rejected. Setting var.domain (e.g. zimmer.tadasant.com) fixes this class at the source by putting a genuine cert on a custom name — while staying reachable only over the tailnet.

The trick is that TLS behind a tailnet is awkward: the firewall opens no public 80/443, so ACME HTTP-01/TLS-ALPN-01 can’t work — only DNS-01 can. On-box renewal would mean parking a Cloudflare token on the droplet, so the work is split so the box holds no DNS credential:

On the droplet (cloud-init.yaml.tftpl, only when var.domain is set): a stock, plugin-less caddy:2 container on :443 that does no ACME. It serves the cert files at /opt/zimmer/certs/{cert,key}.pem and reverse-proxies to the app. The app keeps publishing :80, so the MagicDNS http://… path is unchanged and a Caddy misconfig can’t take the box down. A self-signed placeholder is written at boot so Caddy can start before the real cert arrives.
In CI (scripts/domain-cert.sh, run by domain-cert-staging.yml): discovers the droplet’s tailnet IP, upserts a Cloudflare domain → tailnet IP (100.x) A record, issues/renews the Let’s Encrypt cert via ACME DNS-01 through Cloudflare, pushes only the cert onto the box over tailscale ssh, and restarts Caddy (the Caddyfile sets admin off, so there’s no live-reload endpoint — a restart re-reads the bind-mounted files). The Cloudflare token lives only in GitHub Actions.

The A record points at the tailnet IP, so tailnet peers resolve and reach it while everyone else gets an unroutable address — same tailnet-only exposure as the MagicDNS name, now with a real cert.

Optional Cloudflare edge

Off by default. With it on, the custom domain is served to the internet through a Cloudflare Tunnel, with Cloudflare Access in front of it and Zimmer’s own sign-in behind it. The firewall does not change: the droplet still admits no public TCP at all, because the connector dials out to Cloudflare and serves the hostname over that connection. Tailscale stays for everything that is not a browser. That covers SSH, Kamal deploys, the metrics exporter, and CI’s health checks on http://<tailnet-ip>/up.

It has three parts, and each is off until a deploy turns it on.

1. The connector: `scripts/install-cloudflared.sh`

The deploy runs this converge on every deploy. Given CLOUDFLARE_TUNNEL_TOKEN, it runs the pinned cloudflare/cloudflared image as zimmer-cloudflared, with --network host and restart: unless-stopped, the same shape as Caddy. The token is the one the Cloudflare dashboard gives a remotely-managed tunnel. The script then polls cloudflared’s /ready on host loopback (127.0.0.1:20241), which answers 200 only while a tunnel connection is registered, and fails the deploy if none appears. That check runs on every deploy, including one that changed nothing, so a tunnel that has lost its connections is caught. With the token empty the script changes nothing. ZIMMER_CLOUDFLARED_REMOVE=1 with an empty token is the only thing that takes a connector down. On staging that is the remove_cloudflared input of Deploy staging.

It is idempotent. Same token and same image pin means a no-op that leaves the running tunnel alone. A rotated token, or a bumped IMAGE= in the script, recreates the connector on the next deploy. That is also how a long-lived droplet gets updated. Production reconciles its droplet rather than recreating it, so cloud-init could never deliver this; the deploy does.

The token is deliberately not a Terraform variable. Anything Terraform renders into cloud-init lands in user_data, and user_data is readable from the metadata service by every process on the box, including every agent session. A tunnel token lets whoever holds it run a second connector for the same tunnel, and Cloudflare load-balances real, already-authenticated requests onto it. So the token travels over SSH stdin into /etc/zimmer/cloudflared/token. That file is mode 0400, owned by the image’s nonroot uid, and sits outside /opt/zimmer, which production bind-mounts parts of into the app containers. cloudflared reads it with --token-file, so it never appears in docker inspect. test/infra/cloudflared_test.rb fails the build if a token turns up in main.tf or the cloud-init template, or if any Kamal mount overlaps the token directory.

2. The tunnel’s origin is `http://localhost:8080` (kamal-proxy), not Caddy

Point the tunnel’s public hostname at http://localhost:8080. Of the two candidates, it is the one that keeps the client’s address:

Caddy replaces X-Forwarded-For with the peer it saw. Behind the tunnel, that peer is cloudflared on localhost, so every request would reach Rails looking like it came from the box. That holds for an https://localhost:443 origin as much as for :80, and its cost is concrete: the Quick Router rate limit becomes one bucket shared by every client, and every refusal log line names 127.0.0.1. Caddy can be told to trust its peer, but its Caddyfile is cloud-init-only and never reaches a running droplet.
kamal-proxy appends. With ssl: false, forward_headers defaults on, so the edge’s header survives with kamal-proxy’s docker peer added at the end. TLS terminates at the edge, and the hop from cloudflared to kamal-proxy is host loopback. Health-gated container swaps still apply.

The header shape was checked end to end through a real Cloudflare edge, cloudflared 2026.10.0 and kamal-proxy v0.9.2. A client that sent X-Forwarded-For: 10.9.9.9, 6.6.6.6 arrived as 10.9.9.9, 6.6.6.6,<client>, 172.x.0.1. The edge appends the address it saw, so a forged entry always sits to its left. Rails’ remote_ip takes the right-most entry that is not a private or loopback proxy, which is the edge’s. So the Quick Router rate limit and every “refused from ” log line, including web sign-in’s, key on the real client, with no Cloudflare-specific code.

Zimmer does not read CF-Connecting-IP, on purpose. The tailnet path goes through Caddy, which passes that header through from any tailnet client untouched, so trusting it would let a tailnet peer choose its own address. X-Forwarded-For is safe on both paths: Caddy overwrites it on the tailnet path, and the edge appends to it on the tunnel path. A third path is narrower but real: kamal-proxy publishes :8080 on every interface, so a tailnet peer can skip Caddy and forge X-Forwarded-For itself. That has always been true. It reaches only the tailnet, and Tailscale ACLs decide who that is. test/integration/client_ip_behind_cloudflare_test.rb pins all three cases.

3. Keep the box’s own traffic on the box: `ZIMMER_PIN_DOMAIN_TO_HOST`

Agent sessions call Zimmer at https://<domain>/mcp. Once public DNS points the domain at Cloudflare, that call would leave the box and meet Cloudflare Access, which a session cannot pass. Set ZIMMER_PIN_DOMAIN_TO_HOST=<domain> in the environment kamal deploy runs in, and config/deploy.yml adds --add-host <domain>:<deploy-host-address> to both the web and the worker containers. The address comes from the one populated deploy-host variable (PRODUCTION_HOST or STAGING_HOST), and can be set explicitly with ZIMMER_PIN_DOMAIN_TO_ADDRESS. It must be an IP address. The name then resolves to the host’s tailnet address, where Caddy listens on :443 and serves the domain’s own certificate, so TLS still validates. Unset, nothing is added. Staging sets it on every deploy and then proves it from inside the worker: the name resolves to the staging host, and https://<domain>/up answers 200 through it.

The target is deliberately not Docker’s host-gateway token. Docker resolves that token to the default docker0 gateway, while Kamal attaches app containers to a separate kamal bridge. On a host with no containers on docker0, that interface is down and the pinned connection is refused.

Turning it on

Cloudflare side (not in this repo): create a remotely-managed tunnel, add a public hostname <domain> → http://localhost:8080, and put an Access application on the hostname. Anything that is not a browser needs its own Access policy: a service token, or a bypass for specific paths. That includes REST API clients, remote MCP clients and inbound webhooks.
Deploy side, in the same deploy:
- pass the token to scripts/install-cloudflared.sh <host> as CLOUDFLARE_TUNNEL_TOKEN (staging: the STAGING_CLOUDFLARE_TUNNEL_TOKEN secret, already wired);
- export ZIMMER_PIN_DOMAIN_TO_HOST=<domain> for kamal deploy;
- keep running scripts/domain-cert.sh as before. It reads the live DNS record: when <domain> is a CNAME to *.cfargotunnel.com, it leaves DNS alone instead of upserting the tailnet A record. The upsert would otherwise fail every week. The certificate is still needed, because Caddy keeps serving it to tailnet peers and to the pinned containers. The script’s final check uses --resolve against the tailnet IP, so it tests this box’s Caddy and not the edge.
DNS: once the connector reports registered, let the tunnel’s hostname replace the tailnet A record.

Nothing in the app changes. The hostname stays the same, so APP_HOST, OAuth callbacks and the Host allow-list need nothing. assume_ssl and force_ssl already treat every request as HTTPS, so cookies stay Secure, and the edge sends X-Forwarded-Proto: https regardless.

To turn it off, run the converge once with an empty token and ZIMMER_CLOUDFLARED_REMOVE=1, delete the tunnel’s CNAME (the next domain-cert run puts the tailnet A record back), and unset ZIMMER_PIN_DOMAIN_TO_HOST.

How long a request may take

A request to the web passes up to five hops, and the shortest timeout among them wins:

Hop	Limit	Where it is set
Cloudflare edge (when it is on)	100s, and a Free or Pro zone cannot raise it	Cloudflare
Caddy, the host’s TLS front door	none set (`reverse_proxy localhost:8080`)	`infra/terraform/cloud-init.yaml.tftpl`
kamal-proxy	100s (`proxy.response_timeout`; Kamal’s default is 30)	`config/deploy.yml`
Thruster, in front of Puma	100s (`THRUSTER_HTTP_WRITE_TIMEOUT`; Thruster’s default is 30)	`config/deploy.yml`
Puma	none: a request runs until it finishes, even after a proxy gave up on it	—

The two 100s settings exist for MCP’s sync: true, which holds a tool call open while a session answers: 55 seconds by default, 90 at most. Everything else should still answer well inside 30 seconds. Raising the proxies does not lengthen how long Puma works on a request, because Puma never stopped at the proxy’s limit. It only lets a slow answer reach the client instead of a 504. test/config/kamal_deploy_config_test.rb checks that both settings stay above the sync ceiling.

The Host allow-list

Production and staging turn on Rails’ Host authorization (config.hosts), which refuses a request whose Host or X-Forwarded-Host the deployment does not answer to with an empty 403. That stops DNS rebinding against a tailnet peer’s browser, and it stops a forged Host from reaching anything that builds a URL from the request. AllowedHosts builds the list at boot from the process environment, because config/environments/*.rb runs before the secret store can be reached. Each Host the deployment actually receives, and what admits it:

Who	Host	Admitted by
Browsers via Cloudflare, the host Caddy on `:443`, on-box sessions pinned to the domain	`zimmer.tadasant.com`	the host of `ZIMMER_PROD_BASE_URL` / `ZIMMER_STAGING_BASE_URL`, and of `APP_HOST`
Deploy health checks and smoke tests (`http://<tailnet-ip>/up/deep`, `/`, `/cable` through Caddy `:80`)	`100.x.y.z`	any IP literal, v4 or v6
A person on the tailnet using `http://<tailnet-hostname>/`	`zimmer`, `zimmer.<tailnet>.ts.net`, and `zimmer-1` (Tailscale’s name for a rebuilt droplet while the stale node holds `zimmer`)	`ZIMMER_TAILNET_HOSTNAME`, set in `config/deploy.{production,staging}.yml`
`curl localhost:8080` on the box	`localhost`	always
kamal-proxy’s health gate	the container’s own address	nothing: `/up` is excluded from the check
Anything else, such as a second public hostname		`ZIMMER_ALLOWED_HOSTS`

The list is read from the process environment, not the secret chain, so a base URL set only in the Parameter Store moves the links and the OAuth issuer but not the list. Name that host in ZIMMER_ALLOWED_HOSTS as well.

Admitting every IP literal costs nothing: an IP cannot be a DNS-rebinding target, and Cloudflare only routes hostnames it has a record for. Staging’s domain is not on production’s list, and production’s is not on staging’s.

Break-glass: if a legitimate Host is ever refused, set ZIMMER_ALLOWED_HOSTS=* in the environment kamal deploy runs in and redeploy, and the check is off. On staging that is the ZIMMER_ALLOWED_HOSTS repository variable, which Deploy staging passes through. The production deploy lives in the private companion repo and has to pass it the same way. A deployment that configured no hostname at all also leaves it off, because an allow-list with no domain in it would refuse every real visitor. The refused Host is logged at ERROR as [ActionDispatch::HostAuthorization::DefaultResponseApp] Blocked hosts: …. assume_ssl and force_ssl are unaffected. Host authorization runs before the SSL middleware, so a forged Host gets a 403, not an https redirect to itself.

The iOS app’s host (zimmer-app.tadasant.com) needs no entry here. The edge rewrites its Host to the main domain before the request reaches Rails.

The iOS app’s edge audience

GET /native/access-handoff refuses every Cloudflare Access assertion until ZIMMER_NATIVE_ACCESS_AUD names the native app’s Access audience tag, the edge Terraform’s native_app_access_aud output. See the iOS app. Both Kamal configs read it from the environment kamal deploy runs in, the way ZIMMER_ALLOWED_HOSTS is read. On staging, Deploy staging passes the ZIMMER_NATIVE_ACCESS_AUD repository variable through. The production deploy lives in the private companion repo and has to pass it the same way. It is not a secret.

Background jobs and durable state

config/environments/production.rb sets good_job.execution_mode = :external, which requires a separate bundle exec good_job start process. Kamal runs exactly that as a dedicated worker role (config/deploy.staging.yml), alongside the web role — so cron, pollers, orphan cleanup, token refresh, and catalog refresh all run. The deploy workflow asserts the worker container is up before it reports success.

On staging the worker role also runs under the sysbox-runc runtime as container-root, so agent sessions get their own Docker daemon inside it and can use .agent-containers/. Deploy staging carries a nested_docker dispatch input, on by default; unchecking it deploys the worker under plain runc as uid 1000, which is the rollback. The workflow preflights the droplet for sysbox before the cutover and verifies the running worker after it. Production is unaffected and still defaults off — see Nested Docker for agent sessions.

Both roles mount the same durable named volumes, so state survives a deploy and a container recreate:

zimmer_data → /home/rails/.zimmer — the clones (~/.zimmer/clones), scratch, and every Claude session’s own CLAUDE_CONFIG_DIR (~/.zimmer/claude-config/<session_id>), which is the only credential file a Claude session reads and holds mcpOAuth and nothing else (the DB owns the chain).
claude_home → ~/.claude — Claude Code’s transcripts (projects/), which every per-session config dir symlinks back to. No subscription credential lives here any more.
codex_home → ~/.codex (CODEX_HOME) — Codex’s rollout transcripts, auth.json, and thread store.
pi_home → ~/.pi/agent (PI_CODING_AGENT_DIR) — Pi’s credential and provider state (auth.json, models.json, settings.json). Its transcripts live in the session’s own clone under zimmer_data.
gh_config → ~/.config/gh — the GitHub CLI’s stored auth (from an interactive gh auth login). On staging the durable credential is instead GH_TOKEN, minted for the non-primary tadasant-test account and resolved from the Parameter Store into the process environment on every boot and poll tick — so it survives a rebuild without anyone logging in again. See Staging gh auth.
claude_local → ~/.local — where bin/docker-entrypoint’s background claude update writes.

No host Docker socket is bind-mounted into either role. The docker commands agent sessions and DockerCleanupJob run reach the inner daemon bin/docker-entrypoint starts inside the worker under sysbox — see Nested Docker for agent sessions.

Where a mount is declared, and why that matters

The list above lives once, in config/deploy.yml, on both roles. config/deploy.staging.yml and config/deploy.production.yml carry only what genuinely differs between the two destinations: hosts, the worker’s memory: cap, the nested-Docker runtime/user pair, and the env blocks.

That split is load-bearing, because of how Kamal merges the two files. load_config_files folds the destination file over the base with ActiveSupport’s deep_merge!, which recurses into hashes and replaces arrays. So a destination that restates volume: does not add to the shared list — it overrides it, and the next mount added to the base silently reaches the other destination only. That is the shape of the bug 678f768 had to fix for CODEX_HOME: the runtime home lived on the container layer, every deploy destroyed it, and the next resume failed with “no rollout found”.

To add a mount everywhere, add it to config/deploy.yml. To add one for a single destination, use that destination’s top-level volumes: key — Kamal appends it to every app role’s docker run, so it stacks with the shared list instead of replacing it. Production does exactly that for its three /opt/zimmer bind mounts — the catalog and its encrypted credentials, both delivered by artifacts-sync-prod, and the private extension. test/config/kamal_deploy_config_test.rb asserts the merged result for both destinations, so getting this wrong fails CI rather than a deploy.

Three memory caps, and what each one protects

The worker role carries a container memory: cap — 10g in production, 2g on staging — and it protects the host. Uncapped, an overshoot in the only role that runs arbitrary work is a global out-of-memory event: the kernel takes victims across every cgroup, sshd and Caddy lose their working set, and the droplet stops answering. The cap relocates that kill into one cgroup, at a moment the config chose. Both numbers are derived from their own droplet and are not transferable between them; the reasoning is written out in the deploy files themselves.

Underneath it, ZIMMER_SESSION_MEMORY_MAX_MB gives each agent session its own cgroup and its own memory.max — 4096 in production, 1024 on staging. That one protects the worker’s own cgroup from a single session inside it, which the container cap cannot: one runaway command used to be able to spend the whole budget and let the kernel pick a victim by size rather than by blame (#815). Setting it to 0 disables the bound, which is the break-glass; changing it is a deploy, never a shell.

Read “underneath” literally, because the natural misreading of it is that session memory has moved out of the container’s budget. It has not. cgroup v2 is hierarchical: the per-session cgroups live at zimmer.sessions/sessions/session-<id>, inside the container’s cgroup, so every byte a session charges is charged to the container cap as well. The per-session bound catches one runaway session; it creates no headroom and cannot stop N in-budget sessions from summing over the cap — six sessions at 4 GiB is 24 GiB against 10g, which is why it is a runaway bound and not an admission-control budget.

Between the two, ZIMMER_SESSIONS_MEMORY_MAX_MB caps all sessions together — 6144 in production, 1024 on staging — on a zimmer.sessions/sessions pool cgroup that holds every session cgroup and not the Rails worker. Per-session bounds do not sum to anything: eight sessions each far inside their 4 GiB reached 8.7 GB between them, the container cap fired, and the kernel killed bundle exec good_job start — the worker running every session on the box (#981). The pool’s number is chosen so that it fires before the container cap does, leaving 4096 MB for the worker and the inner dockerd that sit outside it; a pool sized without that margin would be decorative. 0 means no aggregate cap, with per-session bounds left in force.

The pool bounds the blast radius; it does not reduce the demand. A pile-up still exhausts it, and a session still dies — what changes is that the victim is a session process rather than the GoodJob worker that runs all of them. The width of the agents lane (GOOD_JOB_AGENTS_THREADS, 8) remains what limits how often that happens, and config/connection_budget.rb carries the measurements behind that number — the CPU ones that size it, and the memory ones it still has to fit.

The mechanism, the sizing arguments, and what a session sees when either fires are in Each session gets its own memory bound; the deployments that get no bound at all are in Limitations.

Ops actions ship with the deploy

Nothing Zimmer needs done in production requires a shell on the box. A feature is not finished when the code is deployed and an operator still has to SSH in and run something; that step is part of the feature, and it has to ship with it.

There is no fallback here to fall back to. Agent sessions run on the production droplet, the operator key is deliberately not authorized as root there, and the SSH agent root is excluded from the catalog baked into the image — see SSH access. So an ops step that needs a shell is a step no agent can take and a human has to be interrupted for.

Three delivery mechanisms, in order of preference:

A deploy. A migration, or — for a one-time step that needs application code — a post-deploy task in db/post_deploy/. That is the default answer, and it has its own section below. TokenUsageBackfillJob predates it and is the bespoke worked example of the same shape: it starts a sweep on the first tick after the deploy, records its progress in a table, and costs an indexed lookup per tick forever after. See Token spend.
A scheduled idempotent job. Anything that has to keep converging — refreshes, reconciliation sweeps, cleanups. Idempotence is what makes an unattended cron safe to leave running. A recurring need is a cron entry, not a post-deploy task.
The app’s own surfaces — a button in the web UI, a REST endpoint, an MCP action. This is where operator-triggered actions belong. The Costs page’s re-scan button, POST /api/v1/costs/backfill and action_health’s backfill_token_usage are the same request through the three surfaces Zimmer already exposes.

Two obligations come with it. An action that runs unattended must be safe to run repeatedly — for the backfill that is the unique index on request_id, which makes re-ingestion a no-op. And it must give an observable answer to “did it run, and what does it cover”, or an operator has traded a shell for a guess. A rake task is still fine as a developer convenience; it is not the delivery mechanism.

The one sanctioned exception: an action Zimmer must not hold the credential for

There is exactly one case where a rake task is the delivery mechanism, and it is worth writing down so it is applied narrowly rather than reached for: when running the action inside Zimmer would require baking a credential into the image that Zimmer is deliberately built not to have.

The worked case is the Parameter Store namespace migration (Running the migration). Zimmer’s resolver key ships in the image and reaches every session’s environment resolution; it holds three read roles and no write role, and that absence is checked at runtime. Delivering the migration as a post-deploy task would mean shipping the write-capable key alongside it to run once, permanently widening the blast radius of the one baked credential. So the task runs from wherever the writer credential already is, and it touches only Google — no Zimmer database, no running process — so it needs no shell on the box either.

The two obligations above still apply, and are still met. The migration is safe to run repeatedly by construction (it holds no cursor; every step is decided from what the store holds now), and “has it run, and what does it cover” is answered on the Connectors page, which needs no shell. An exception to the mechanism is not an exception to the properties the mechanism exists to guarantee — if you cannot state both, this is not your case.

Dropping a column takes two deploys

A migration that drops a column and the code change that stops using it cannot ship in the same deploy. kamal-proxy health-gates the cutover, so the old and new containers run together until the new one answers /up. That is the same window the connection budget doubles for, and bin/docker-entrypoint has already run db:prepare by the time it opens. So for its whole length the old processes are serving against the new schema.

That is not a survivable combination. The old process booted with the column present, so its model still defines the attribute; its SELECTs now come back without it, and reading the attribute raises ActiveModel::MissingAttributeError.

Dropping sessions.blocked_by_session_id in one phase did exactly this: 12 ERROR records in 12.8 seconds across the three GitHub pollers of the day — since fused into GithubPrPollPassJob — which crossed the backend log-error alert threshold and paged #alerts. The polls it interrupted really did abort — they recovered on the next tick, but a PR-merge notification arrived a poll interval late. See zimmer#482.

The recipe

Deploy 1 — stop reading the column. Add it to the owning model’s ignored_columns and delete every code reference: the attribute, the association, the scope, the strong parameter, the view, the fixture column. No migration.

class Session < ApplicationRecord
  # Phase 1 of dropping this column. Active Record stops loading it, so nothing
  # in either image reads it. Phase 2 drops the column and this line.
  self.ignored_columns += %w[blocked_by_session_id]
end

ignored_columns makes Active Record behave as if the column were already gone — no attribute method, and it is left out of SELECT lists. That is what makes the next deploy safe: by the time the column disappears, the old containers were never reading it either.

Deploy 2 — drop it. A separate pull request, merged after deploy 1 is actually live. It drops the column and removes the ignored_columns line, and it annotates the migration with the number of the pull request that shipped phase 1:

# two-phase-drop: phase 2 of #474
class DropBlockedBySessionFromSessions < ActiveRecord::Migration[8.0]
  def up
    remove_reference :sessions, :blocked_by_session, index: true
  end
end

Leaving the ignored_columns entry behind is not harmless: it silently hides a column of that name if one is ever added back.

The guard

TwoPhaseColumnDropGuard (test/support/two_phase_column_drop_guard.rb) fails CI when a migration makes a column or a table stop answering to the name the running image compiled against, in the forward direction, without the annotation its shape calls for. It sorts what it finds into four shapes:

Shape	Caught in	Annotation
`column_drop`	`remove_column`, `remove_columns`, `remove_reference`, `remove_belongs_to`, `t.remove`, raw `DROP COLUMN`	`# two-phase-drop: phase 2 of <ref>`
`column_rename`	`rename_column`, `t.rename`, raw `RENAME COLUMN`	`# expand-contract: contract of <ref>`
`table_rename`	`rename_table`, raw `RENAME TO`	`# expand-contract: contract of <ref>`
`table_drop`	`drop_table`, raw `DROP TABLE`	`# expand-contract: contract of <ref>`

Raw SQL is matched against string contents, so heredocs count — which is how anyone actually writes SQL in a migration. A rename is only recorded while the statement in scope alters a table, so the ALTER INDEX … RENAME TO that ends a zero-downtime index swap does not trip it, and neither does ALTER TYPE or ALTER SEQUENCE. rename_index and remove_index are absent for the same reason: an index carries no attribute and no query of the old container names it, so neither strands anything. Lines beginning -- are SQL comments, not statements, and are skipped.

The two annotations are separate because they assert different facts. two-phase-drop says an ignored_columns deploy shipped. expand-contract says an earlier deploy left the old name unread and unwritten by the running image — which for a rename is the end of an add-dual-write-backfill-switch sequence, not a one-line model change. Neither annotation clears the other’s shape, so a migration that renames one column and drops another has to carry both.

It parses the migration rather than grepping it, because the direction is a syntactic fact. A remove_column inside def down, dir.down { } or revert { } is the undo of an add_column, and a regex cannot tell it from the forward body two lines above: 14 of this repo’s migrations contain a remove_column and only 6 of them actually drop one. The pruning is by method name though, so a removal factored out of down into a helper is still reported. Inline it into down rather than annotating a phase 1 that never happened.

The annotation is the escape hatch, and it has to name something a reviewer can go and read: a PR or issue number (#474), a commit sha, or the phase-1 migration’s version. It is read off the parsed comments, so the same text inside a SQL heredoc is not evidence, and # two-phase-drop: phase 2 of the earlier PR does not pass.

Two jobs run it. bin/rails test picks up test/migrations/two_phase_column_drop_test.rb in test-unit, which is also where the ignored_columns half is checked — an entry naming a column that no longer exists is a phase-2 cleanup someone forgot. The guard itself needs neither Rails nor a database, so lint runs it directly and answers in seconds. That is also how you run it by hand:

bundle exec ruby -r./test/support/two_phase_column_drop_guard -e 'puts TwoPhaseColumnDropGuard.report'

lint runs a second guard of the same shape for the same reason — InertSubprocessTimeoutGuard, which fails when a Timeout.timeout encloses an Open3 call that joins its wait thread in an ensure. It is documented in background-jobs.md.

Ten migrations that shipped before the guard covered their shape are named in its GRANDFATHERED list — seven single-phase column drops, and the three single-phase renames that were invisible to it until the detected set widened. That list is closed, and a GRANDFATHER_CUTOFF assertion keeps it that way: a new drop or rename gets the deploys and the annotation, not an eleventh entry. The newest drop is the case for the guard — #680 dropped app_settings.provenance_via_mcp_enabled in a single phase on 2026-08-28, twelve days after the incident, because nothing was checking. The newest rename is the case for widening it: #629 renamed two app_settings columns on 2026-08-23 and nothing said a word.

What it does not cover. Schema changes that keep every name intact — a widened type, a changed default, a new NOT NULL — are outside it. Those can still break an old container, but they break it on the values rather than on the names, and the guard reads names.

Renames and table drops expand before they contract

The mechanism above was never about dropping a column. It is that kamal-proxy health-gates the cutover, so the old containers keep serving after db:prepare has already changed the schema. Anything that makes a column or a table stop answering to the name the old image compiled against reproduces it, and three shapes do:

Migration	What the old container hits
`rename_column`	`ActiveModel::MissingAttributeError` on reads — the `SELECT` comes back with the new name and without the old one — and `PG::UndefinedColumn` on writes, because its `INSERT` still names the old column.
`rename_table`	`PG::UndefinedTable` on every query against the table, not one attribute on one model.
`drop_table`	`PG::UndefinedTable`, the same, permanently.

A rename looks like one operation and is two: a drop and an add at the same instant. So it does not get a two-deploy recipe — it gets the drop’s recipe with an add in front of it.

Renaming a column

Nothing renames in place. Expand: add the new name and fill it. Contract: take the old one away once nothing touches it.

Deploy	Migration	Code
1 — expand	`add_column :sessions, :goal, :string` (nullable, no default), `change_column_null :sessions, :stop_condition, true` if the old column was `NOT NULL`, plus a post-deploy task that backfills	the model writes both names on every write; reads still come from `stop_condition`
2 — switch reads	none	every read moves to `goal`; both are still written
3 — stop writing the old name	none	`self.ignored_columns += %w[stop_condition]`, and the dual-write goes
4 — contract	`remove_column :sessions, :stop_condition`, annotated `# two-phase-drop: phase 2 of #<deploy-3 PR>`	the `ignored_columns` line goes

Deploys 3 and 4 are exactly the two-phase drop, because by then that is all that is left. Deploys 1 and 2 are what a rename adds.

The backfill is a post-deploy task, not part of the migration. A migration runs in bin/docker-entrypoint, before the new container answers /up — so every row the old containers write during the swap window lands after the backfill has already read the table, with the new column NULL. A post-deploy task runs from the new image a couple of minutes later, once the old containers have drained. Nothing can slip past it in the other direction either: from deploy 1 onward every write from a new container fills both names.

class BackfillSessionGoal < PostDeployTask
  def up
    sweep(Session.where(goal: nil).where.not(stop_condition: nil)) do |batch|
      Session.where(id: batch.map(&:id)).update_all("goal = stop_condition")
    end
  end
end

sweep pages by primary key and checkpoints its cursor, so the task resumes where it left off if a slice runs out of its 90-second budget. The goal IS NULL predicate is what makes a second run a no-op.

The old column has to be nullable before deploy 3, whatever else you do. ignored_columns leaves it out of every INSERT, so a NOT NULL column with no database default gives PG::NotNullViolation on the new image’s first write. Relax the constraint back in deploy 1, where it cannot hurt anyone: the old containers still write the column, and dropping a constraint never breaks a writer.

Deploys 2 and 3 are separate for a reason. Merge them and, for the length of that swap window, the new containers are inserting rows with the old column NULL while the old containers are still reading it. You may merge them when the old code shrugs at a NULL there — a display string, say — and you may not when it does not.

The cheapest correct rename is usually no rename. Four deploys to improve a name is a real price. alias_attribute, or just living with stop_condition in the schema and goal in the code, costs nothing and strands nobody. Rename when the old name is actively misleading, not when the new one is nicer.

Retyping a column: the shadow takes the old name

Changing a column’s type is a rename wearing a different hat, and sessions’ five json → jsonb columns are the worked example (#847). The obvious migration is the one not to write:

ALTER TABLE sessions ALTER COLUMN metadata TYPE jsonb USING metadata::jsonb;  -- DO NOT

json and jsonb are not binary-coercible, so that rewrites the whole table under ACCESS EXCLUSIVE — transcript and all, which SessionContentSearch puts at gigabytes. So it goes through a shadow instead, and the twist is in the contract: the new column does not keep its scaffolding name, it takes the old one, and every reader in the app lands on the new type without a single call site changing.

Deploy	Migration	Code
1 — expand (#1018)	`add_column :sessions, :metadata_jsonb, :jsonb` — catalog-only, plus a post-deploy task that backfills it	every write path fills both; nothing reads the shadow
2 — swap (#1179)	converge the shadow, `rename_column :metadata, :metadata_json_legacy`, `rename_column :metadata_jsonb, :metadata`, then `add_column :metadata_jsonb` again, empty; rebuild the expression indexes the first rename carried away	the dual-write goes; all ten dead names go into `ignored_columns`
3 — contract	`remove_column` on the ten dead names, annotated `# two-phase-drop: phase 2 of #1179` — merged only once deploy 2 was live	the `ignored_columns` line goes

Three things in deploy 2 are what make it survivable, and each one is a way the obvious version breaks:

Re-adding the shadow name, empty. The old containers write it from every path the dual-write covered, and columns_hash was cached at boot — so they will name that column whether or not it is still there. Drop it and every save from an old container is a PG::UndefinedColumn for the length of the swap window.
Renaming the original aside instead of dropping it. Dropping it here would be a single-phase drop: no image ever shipped with it in ignored_columns, so the two-phase-drop annotation would be a claim about a deploy that never happened. Renaming retires the name, keeps the values for one deploy as the undo, and makes deploy 3’s annotation true.
Converging the shadow first, behind an explicit LOCK TABLE. A backfill that reads succeeded proves the rows that existed when it ran, not the rows as they are now — a writer reaching the original through update_all or raw SQL leaves a shadow that is stale and not null, and the rename would promote it silently. So deploy 2 re-runs the predicate as an UPDATE … WHERE <shadow> IS DISTINCT FROM <original>::jsonb. Take ACCESS EXCLUSIVE explicitly, as the first statement of up: an UPDATE on its own takes only ROW EXCLUSIVE, so without it a one-sided writer can commit in the gap between the convergence and the first DDL statement, and the rename promotes exactly the stale shadow the convergence was there to catch. Taking the strongest lock up front instead of upgrading into it mid-transaction also removes the lock-upgrade deadlock the sequence would otherwise be shaped like.

The old containers read the swapped column happily, which is the part that makes the whole shape work: Active Record casts json and jsonb to the same Ruby Hash, the wire format is text either way, and json → jsonb is an assignment cast in PostgreSQL — so even a write that explicitly casts its expression back to ::json lands in the retyped column without complaint. A retype that is not assignment-castable in the old direction does not get this recipe.

rename_column carries its indexes with it. The expression and partial indexes over metadata followed it to metadata_json_legacy, names included, so recreating them against the new column collides until the old ones are dropped. remove_index then add_index in the same migration — an index carries no attribute, so neither call strands anything and the guard ignores both.

Renaming or dropping a table

A rename_table is the same expand-and-contract one level up: create the new table, write both, backfill from a post-deploy task, switch reads, stop writing the old table, drop it. That is a lot of machinery for a name — self.table_name on the model lets the class say Goal while the table stays stop_conditions, and it ships in one deploy.

A drop_table is genuinely two deploys, because there is nothing to expand into:

Deploy 1 — stop querying it. Delete the model and every reference to it: the association, the fixture, the dashboard entry, the job, the route. No migration. Nothing in either image then touches the table.
Deploy 2 — drop it. A later pull request, merged after deploy 1 is live, carrying the annotation.

The annotation

The three name-changing shapes take their own annotation, and it names the deploy that left the old name unread and unwritten:

# expand-contract: contract of #474
class DropWidgetsTable < ActiveRecord::Migration[8.0]
  def up
    drop_table :widgets
  end
end

It is deliberately not the two-phase-drop one. That annotation claims an ignored_columns deploy shipped; this one claims a whole expand-and-switch shipped, and a reviewer checking the wrong claim against the wrong pull request would find nothing wrong. Neither clears the other’s shape, so a migration that renames one column and drops another carries both. Same rule on the reference: a PR or issue number, a commit sha, or the earlier migration’s version — something a reviewer can go and read.

On a rename_column the annotation should be rare to the point of suspicious. Follow the recipe and there is no rename_column left to annotate — deploy 4 is a remove_column carrying two-phase-drop. It is honest only when the old name is already dead in the deployed image: a column nothing has read since some earlier release, renamed for tidiness.

Rails builds its migration list at boot, not at migrate time, and it raises rather than picking a winner: two files whose version prefixes are equal give ActiveRecord::DuplicateMigrationVersionError, two files declaring the same class name give DuplicateMigrationNameError. maintain_test_schema! walks that list on the way into every test run, so the failure is not one broken test — it is the whole suite dying before any test executes.

That shipped on 2026-09-05. Two PRs each hand-wrote a migration numbered 20260905180000 — #1005 against trigger_conditions, #1008 against sessions. Both were green in isolation, because the collision did not exist on either branch. It existed on main the moment the second one merged, and from then on test-unit and test-system failed identically on every commit until the second migration was renumbered — for that window, no commit on main was provably green.

MigrationVersionGuard (test/support/migration_version_guard.rb) reports both collisions, naming the version or class name and every file claiming it. It reads filenames and nothing else — not even the file bodies the two-phase guard parses — so like that guard it needs no Rails, and lint runs it with no Postgres and no Redis. That is the point: what it detects is a Rails boot failure, so a check that had to boot Rails would be silent in the case that matters most. It answers in under a fifth of a second over this repo’s migrations. test/migrations/migration_version_test.rb runs it again inside bin/rails test, and by hand it is:

bundle exec ruby -r./test/support/migration_version_guard -e 'puts MigrationVersionGuard.report'

It carries a copy of Active Record’s own MigrationFilenameRegexp and glob rather than a stricter pattern of its own, and it compares what Rails compares: the version as an Integer, the name camelized. Every one of those matters, because each difference is a collision that hides. A guard that read 14 digits off a flat directory listing would wave through a 13-digit version, a leading zero, an adapter-scoped ..._create_foo.postgresql.rb and a migration in a subdirectory; one that compared the snake_cased names would wave through add_widget_2 alongside add_widget2, which are one AddWidget2 to Rails. Tests assert the copied regexp still matches the constant and that the guard reads every .rb file in db/migrate, since it cannot ask Active Record at the moment it runs.

Generating migrations with bin/rails generate migration rather than hand-writing the timestamp avoids the collision within a branch. Renumbering after the fact means re-dumping db/schema.rb too, so its version: still matches the newest migration on disk — SchemaDumpTest asserts that.

What it does not cover. Duplicates only. Rails raises IllegalMigrationNameError on a file the glob matches and the regexp does not — 20260101000000_AddFoo.rb — at the same point in boot, and the guard says nothing about it: one file is enough to trigger it, so there is no pair to report, and the fix is a different one. Nor does it look at db/cable_migrate, the cable database’s configured migrations path, which is not a directory in this repo.

More importantly, a branch-local check sees a duplicate that exists on that branch. Two independent branches each adding …180000 each pass, which is exactly how the incident happened — and how it happened again on 2026-09-11, when #1155 and #1156 both hand-wrote 20260912120000 and #1163 renumbered the second one.

Closing it takes a repository setting, not a check this repo can ship: require branches to be up to date with main before merging, or put a merge queue in front of it, so the collision is on the branch by the time CI runs. Neither is configured here — main is not a protected branch and no workflow has a merge_group trigger.

Reading origin/main’s migrations alongside the branch’s would narrow the window, not close it, and the 2026-09-11 recurrence is the proof: lint last ran on #1156’s head at 19:05 UTC, three minutes before #1155 put the colliding migration on main, and nothing re-ran it in the 59 minutes before #1156 merged. A union check catches the second PR only when CI happens to run after the first one lands, which is exactly what no setting currently guarantees.

Renumbering safely

Renumber the migration that landed second. The one that landed first held the version alone for a while, so a database that migrated in that window recorded the shared version for it; renumbering that one would leave such a database skipping the other migration forever and re-running the first against a column it already added.

That reasoning covers db:migrate, which validates duplicates in Migrator#initialize and so applies nothing at all while the collision is live. It does not cover a schema load. Loading db/schema.rb into a database that does not exist yet ends in assume_migrated_upto_version, which validates duplicates only over versions strictly below the schema version — and a collision at the newest migration is not below it. So a database created during the window comes up perfectly cleanly, with both migrations’ columns present (the dumped schema has them) and a single row standing in for both versions. The renumbered migration is then pending against it, and the column it adds is already there.

Give the renumbered migration if_not_exists: whenever a database could have been created that way, which is any window longer than a moment: every agent-session clone builds its dev and test databases by loading the schema. add_column, add_index, add_foreign_key, add_check_constraint and create_table take that option; anything else — a change_column_null, a raw execute — needs a column_exists? guard written by hand.

And it happened again, with a bigger blast radius

On 2026-09-11 #1155 and #1156 both merged a migration numbered 20260912120000, and #1163 renumbered the second an hour later. What made this one worse than the 2026-09-05 collision was not the collision — it was what the new code did while the collision was live.

ActiveRecord::Migrator validates duplicates in #initialize, so db:migrate raised before applying anything. Production ran the new image against a database that consequently had no api_keys.grant column, and ApiKey.authenticate read that attribute on every authenticated request. The result was NoMethodError: undefined method 'grant' — a 500 on every REST API and /mcp call, which is every agent session in the fleet, while the web UI kept answering 200 because its pages never touch that table. A missing key still returned 401, and a valid one 500ed.

So: code may not hard-require a column its own deploy adds. The two-deploy rule for dropping a column exists because old containers keep serving after the schema moves; this is the mirror image, and it bites whenever a migration has not run yet for any reason — a duplicate version, a failed migrate step, a rollback, a container that boots before the migrate step finishes. Read a newly added column through one accessor that answers the pre-migration default when the attribute is absent, the way ApiKey#effective_grant does:

def effective_grant
  has_attribute?(:grant) ? self[:grant] : API_GRANT
end

column_names is the table’s own shape, so the answer is the meaning that row already had. Ask the table rather than the row: has_attribute? is also false for a record loaded through a projection that left the column out, and answering the old default for one of those would apply it to a row that really does carry a value. Guard the column’s validations with the same test, and have any write that needs the column fail loudly rather than silently storing something weaker.

The default is only correct in the forward direction, and that bound is the price. It holds while the column has never existed, because no row can yet carry a value other than the default. It does not hold in reverse: roll the migration back, or run an older image after narrow values exist, and every one of them silently reads as the default — for api_keys.grant that means every Quick Router key becomes a full-API key. So the accessor is not a licence to treat the migration as optional; it buys the window between the code landing and the migration applying, and nothing after it. Say so wherever the reverse is plausible, and prefer an irreversible down on a migration whose column carries a privilege.

There is no guard for any of this: the discipline is to route every read of a new column through a single accessor in the deploy that adds it — including the ones that do not look like reads, such as an Administrate dashboard, which renders ATTRIBUTE_TYPES by public_send and will raise on the raw attribute exactly as application code does.

One-time post-deploy tasks

If the step runs once and then never again, write a post-deploy task. This is Zimmer’s equivalent of the after_party gem, and it is the mechanism the rule above points at: you should not have to invent the apparatus each time, and you should not have to reach for a shell.

A task is one file in db/post_deploy/, named like a migration:

bin/rails generate post_deploy_task prune_orphaned_widgets
# => db/post_deploy/20260830100500_prune_orphaned_widgets.rb

class PruneOrphanedWidgets < PostDeployTask
  def up
    Widget.where(owner_id: nil).delete_all
  end
end

That is the whole authoring surface. Nothing registers it, nothing else has to be edited, and PostDeployTaskJob — a two-minute cron entry on the default queue — picks it up within a couple of minutes of the deploy. It runs at priority -100, ahead of ordinary default-queue work. A database migration also promotes any safe, inherited runner row to that priority: Active Job stores priority when it enqueues a row, so changing the job class alone cannot update a singleton row left by the previous release. Together those safeguards keep a default backlog from blocking the task that must migrate that backlog.

What it guarantees, and what it does not


Runs once	`succeeded` is terminal in `post_deploy_task_runs`, keyed on the file’s timestamp. Renaming the class does not re-run it.
Never runs twice at once	The ledger row is claimed with a conditional `UPDATE`, so two containers coming up together produce one winner and one no-op.
Never wedges the deploy	Nothing in the deploy waits on it. It is a cron job in the worker, not an entrypoint step or a Kamal hook, precisely so that a task which is slow or raises cannot hold the cutover.
Never wedges the next task	Each task is worked inside its own rescue. A failure is recorded and the task behind it still runs.
*Does not* guarantee idempotency**	The mechanism cannot know whether a task that died halfway half-applied. Write `up` so that running it twice is harmless — an upsert, a `delete_all` of a shrinking set, a `WHERE … IS NULL` guard — exactly as you would a data migration.

Long-running work

This is for the hour-long case as well as the ten-millisecond one. Each task is handed a 90-second budget (PostDeployTaskJob::SLICE_BUDGET). A task that cannot finish inside it returns PostDeployTask::CONTINUE and is resumed on the next tick from the cursor it saved. sweep is that loop packaged — it walks a relation in key order, checkpoints after each batch, and yields the worker when the budget runs out:

class PruneOrphanedWidgets < PostDeployTask
  def up
    sweep(Widget.where(owner_id: nil), batch_size: 500) do |batch|
      Widget.where(id: batch.map(&:id)).delete_all
      checkpoint!(deleted: stats.fetch("deleted", 0) + batch.size)
    end
  end
end

The token-usage backfill’s hour of wall clock is exactly this shape; it predates the mechanism and keeps its own apparatus, but a new task of that size does not need one.

Two things to know about the budget. It is checked between batches, not inside one, so a single batch query runs to completion however long it takes — pass sweep a relation the database can serve from an index, or the last query of the sweep (the one that proves nothing is left) is a full scan. And version order is honoured, but only a failure is not a barrier: a task that keeps yielding is still the earliest due task on the next tick, so one written to run for hours delays the tasks behind it by hours.

When it fails

A task that raises is recorded on its row and retried with backoff — 1m, 5m, 15m, 30m, 1h — and then stops, so a durably broken task is not burning a worker slice every two minutes. At that point it reads as blocked, and the health page turns critical. “Raises” is wide on purpose: it covers ScriptError as well as StandardError, because a task file that will not load (SyntaxError, LoadError) and one generated but not yet filled in (NotImplementedError) are both the former. A worker killed mid-task (a deploy, an OOM) leaves its claim behind; the lease expires after 20 minutes and the abandoned claim is converted into an ordinary failure, so it retries down the same path.

A worked example: the gate decision backfill

ImportGateDecisionLedgers is the shape most non-trivial tasks take. It has to move 1,469 entries across 19 JSON files in a different repository into gate_decisions, which means fetching a few megabytes over the network — so it slices by file, checkpoints the names of the ones it finished into the cursor, and returns CONTINUE when the budget runs out. It is idempotent because it only ever inserts, keyed on each entry’s identity; a second pass finds every row already present and writes nothing. Its stats carry the per-file counts, which is what makes “did it import everything?” a question the health panel answers rather than one somebody has to take on trust. See the gate decision ledger.

ImportWorkBacklog is the small end of the same shape: one ~100 KB file, 117 items, one slice, keyed on each item’s id so a second pass writes nothing, with the counts on its stats. See the work backlog.

ImportGateDecisionsAppendedAfterTheLedgerImport is the same importer run over a pinned set, which is the shape a task takes when the source has changed since the first import and the table cannot be corrected. The archive grew by 52 entries after the first import read it. Two of them were also recorded live under a different key, and gate_decisions is append-only. So the task names the 50 it imports by key and verdict and inserts nothing else. A pinned entry that has since been recorded live is skipped, and one it cannot find fails the run rather than letting it report succeeded. See the appends the first import missed.

ScopeExistingCloneEnvFiles is the shape a task takes when a change alters what the app writes to disk rather than to the database. Scoping a clone’s .env to its session’s artifacts (what reaches a session clone’s .env) applies from the next prepare, so a clone sitting idle, parked or archived would have kept the whole bundle until its session next ran. The task walks every session with a recorded clone, rewrites the .env of each one still on disk through the same SessionEnvFile the prepare path uses, and leaves clones without a .env alone. It is idempotent because the rewrite is a pure function of the session’s selection, so a second pass writes the same bytes; its stats count the files it rewrote, the sessions it found with no .env on disk, and the ones that failed, by id.

RearmWakesBrickedByUnresolvableAgentRoot is the other shape a one-time task takes: not an import but a repair of live rows, where selecting one row too many is as bad as selecting one too few. Two things follow from that. Its idempotency is structural rather than keyed — re-enabling a trigger moves it off failed, which is the first term of its own predicate, so a second pass matches nothing and it can never re-arm a row a human has since cleared. And because the write itself destroys the evidence (Trigger#clear_failure_state_when_leaving_failed sheds failed_at and last_error on the way out of failed), it logs each row whole before mutating it and puts a bounded digest in stats. A task whose effect is one-way owes a reader that much: the log is the durable copy, stats is the copy reachable without a shell, and neither is the row it changed. See re-arming bricked wakes.

SweepMisrecordedAgentPostedGithubComments is that repair shape taken to its sharpest: it deletes rows, one-way, from a table nothing else can reconstruct. What makes that safe to run unattended is not care, it is asymmetry. Deleting a row that was correct hands an agent’s own GitHub comment back to every session tracking the PR; leaving a wrong one costs exactly what the status quo already costs. So the task is written to be too narrow rather than too broad: a row is deleted only when the fixed classifier says it is not a post and the permalink is still in the output of a command that names a posting invocation, which is the only kind of result the old classifier could have read it from — and every row the signal cannot reach — no session, no transcript, a transcript that parses to nothing — is kept and counted under its own reason. That counting is the other half of the design: a sweep with nothing to do and a sweep that silently reached nothing look identical from outside, so stats carries rows_examined, rows_reachable, rows_unreachable, rows_deleted and a kept_by_reason histogram, and each deletion is logged whole — with the command whose output was misread — before the row goes.

BackfillSessionTranscriptChunks is the third shape: a storage move, where the task is copying data it is then going to free. Every session’s transcript has to leave sessions.transcript for session_transcript_chunks (#110), and the whole risk is in the order of those two steps. So the copy is verified before the source is released: per row, in one transaction, it writes the chunks, reads them back, and requires the concatenation to equal the source byte for byte — and, for the legacy Array format, to re-parse to the same events — before it NULLs the column. A row that fails verification has its chunks deleted and its column left alone, so it goes on reading exactly as it did, and is named individually in unmigrated_session_ids alongside the verification_failures count. That list is what phase 2 gates on, not this task’s succeeded — a skipped row leaves the run green with its column still full, and sweep never revisits it, so treating green as “the column is empty” would drop exactly the transcripts the verification saved. Its idempotency is structural for the same reason as the repair above: every row it migrates ends with transcript NULL, which is the negation of its own predicate. And nothing waits on it — Session#transcript reads the legacy column for any row it has not reached yet, so the system is whole from the moment the deploy lands and only gets cheaper as the task walks the table. A read path that is correct only after a backfill finishes is a half-migrated read path. See Where a transcript is stored.

Seeing it without a shell

One object, PostDeployTaskRun.summary, rendered four ways so they cannot disagree:

Surface	Where
Health page panel	`/health` → Post-Deploy Tasks — status per task, counters, the error text, and a Re-arm and run now button. Like the rest of the web UI, neither the panel nor the button asks for a credential
REST	`GET /api/v1/health` → `health_report.post_deploy_task_health`; `POST /api/v1/health/run_post_deploy_tasks` to re-arm
MCP	`get_system_health` reports it; `action_health` with `action: "run_post_deploy_tasks"` re-arms
Supervisor	`/supervisor/post_deploy_task_runs` — read-only, row-level: cursor, stats, lease holder, backtrace

Re-arming is the answer to “the task failed for a reason I have now fixed”. It clears the failure count and queues a pass; it will not re-open a task that already succeeded, because re-running one of those is what writing a new task file is for.

The database connection budget

Managed Postgres hands out a hard, small number of connection slots, and an ActiveRecord pool is a promise to use up to that many. The promise is lazy — an overcommitted app looks perfectly healthy until real traffic calls it in, and then Postgres refuses with FATAL: remaining connection slots are reserved for roles with the SUPERUSER attribute, which Rails serves as a 500. Nothing in a Rails boot compares the promise against the server, so Zimmer does it in two places instead.

config/connection_budget.rb is the single source of truth. Two roles times two databases is four ActiveRecord pools, and they have four different right answers — a flat number is correct for at most one of them:

Pool	Sized for	Why
`web` → `primary`	Puma’s threads (`RAILS_MAX_THREADS`, 3)	A request holds a connection for the request.
`worker` → `primary`	every GoodJob scheduler thread	An executing job holds its connection for the whole job: GoodJob’s advisory lock is session-scoped, so it leases the connection stickily. Zimmer’s agent jobs run for hours.
both → `cable`	concurrent in-flight broadcasts	`solid_cable` takes no advisory lock, so Rails leases per `INSERT` and hands the connection straight back. A broadcast from an hours-long agent job holds one for the write, not for the job.

The same file derives GoodJob’s queue string (config/environments/*.rb read it), so raising GOOD_JOB_AGENTS_THREADS moves the pool that has to serve those threads along with it. They cannot drift apart.

A deploy is the peak, not the steady state. kamal-proxy health-gates the cutover by running the old and new containers together until the new one answers /up — so for that window every connection exists twice. The budget multiplies by two for exactly this reason; a configuration that only fits at steady state turns each deploy into a coin flip.

ConnectionBudget.required_backends is the resulting number. infra/terraform/main.tf refuses to plan against a managed cluster whose plan cannot serve it (DigitalOcean allots 25 connections per GiB of RAM, minus 3 reserved, and the ceiling is not otherwise tunable), and test/config/connection_budget_test.rb asserts the Terraform default still equals the Ruby derivation. To see the whole picture against a live server:

bin/rails db:connection_budget

It prints both halves — what the app commits to, what the server can serve — and exits non-zero when the first exceeds the second.

The Docker images

Dockerfile.base → ghcr.io/tadasant/zimmer-base — the heavy one. Two different workflows publish it, and confusing them is a live trap — see Two paths rebuild the base image. From ruby:3.4.6-slim, it bakes in:

Gems, pre-bundled to /usr/local/bundle with bootsnap precompiled
Node.js 22, the Docker CLI, gh, the 1Password CLI, uv/uvx
Playwright + Chromium and Puppeteer + Chrome (for browser-automation MCP servers)
The npm and Python MCP packages listed in mcp.json (bin/preinstall-mcp-packages)
The AIR CLI @pulsemcp/[email protected] + adapters → /opt/air-cli
The Codex CLI @openai/[email protected] and Claude Code (via claude.ai/install.sh)
The Pi CLI @earendil-works/[email protected], and the three Pi extensions Zimmer loads into every Pi session → /opt/pi-extensions (see Agent harness)

Dockerfile → ghcr.io/tadasant/zimmer — the app image. Copies the app onto the base, re-runs bundle install (which catches Gemfile drift against the base), precompiles assets, drops to USER 1000:1000, and runs bin/thrust bin/rails server.

The app image knows which commit it is

The app image carries the commit it was built from, so a log record can say which deploy it came from without anyone cross-referencing this workflow’s job timings. release-image.yml (all three build attempts) and deploy-staging.yml pass GIT_SHA=<commit> to docker/build-push-action; ARG GIT_SHA / ENV ZIMMER_GIT_SHA=${GIT_SHA} in the Dockerfile carries it into the container; OtelLogsExporter ships it as the service.version resource attribute on every exported log record (Observability).

Two details are deliberate. The ENV sits at the very end of the final stage: an ARG produces no layer, so a changed value invalidates the cache at its first use rather than at its declaration — and this value changes on every commit, so an ENV placed above the RUN steps would rebuild all of them on every build. And the ARG’s default is the empty string, not a placeholder — a hand-run docker build bakes in nothing, and the exporter then omits service.version rather than shipping a blank one. test/config/image_build_workflows_test.rb asserts both ends of that wire and the ENV’s position, because renaming either end alone — or hoisting the ENV — leaves every build green while doing real damage silently.

Two paths rebuild the base image

Nothing depends on a human remembering to rebuild the base, and a base image is not stale merely because the “Build base image” workflow has not run since a Dockerfile.base change. The two paths answer different questions. (A third, deploy-staging.yml, builds its own zimmer-base:staging and never touches either of these tags.)

release-image.yml — the repo declared something new. On every push to main it hashes the five repo inputs Dockerfile.base consumes — Dockerfile.base, Gemfile, Gemfile.lock, mcp.json, bin/preinstall-mcp-packages — into a content key, and looks for zimmer-base:content-<key>. Absent means that exact declaration has never been published, so it builds and pushes the base itself before building the app image FROM that tag. Present means reuse. So a Dockerfile.base change reaches production on the merge that lands it, automatically, and a base build that failed cannot be skipped by a later commit — the content tag is still absent.

build-base-image.yml — the world outside the repo changed. A patched ruby:3.4.6-slim, an apt security update, a newer claude from claude.ai/install.sh. None of that moves the content key, so release-image.yml cannot see it; the monthly cron (0 6 1 * *) and manual dispatch exist for exactly this. It computes the key the same way release-image.yml does and promotes to content-<key> and :latest, because a refresh published anywhere else is one the app image never picks up. It builds uncached for the same reason: the layers a refresh exists to renew are precisely the ones whose cache keys have not moved, so a cache hit replays the image it was meant to replace.

Build, verify, then promote — in that order. The refresh pushes first to a scratch tag (zimmer-base:unverified) that nothing resolves. Only after the image has been pulled back from the registry and checked does a manifest copy promote it onto content-<key> and :latest. The ordering is load-bearing rather than tidy: release-image.yml treats the presence of a content tag as proof that declaration built successfully and will not rebuild it, so an unverified image written there would be served to every app image until a repo input happened to change.

The check itself (.github/scripts/verify-base-image.sh) runs inside the pulled image and asserts two things Dockerfile.base declares — that every name@version it pins resolves at that version, and that every path in its test -f smoke checks exists. test -f at build time says a layer was produced; only this says the tag the release path resolves actually serves it. It refuses an empty expectation list, so it cannot pass by checking nothing.

Promotion is gated on refs/heads/main. A dispatch from any other ref builds and verifies and then stops, leaving the shared tags alone — which is what makes the workflow safe to exercise from a feature branch, and a base image built from unreviewed code is worse than a stale one.

The docs never ship in the image

This documentation site is single-source. The only copy that exists is docs/ in the repository, built by Cloudflare Pages and served at docs.zimmer.tadasant.com. A second copy bundled into ghcr.io/tadasant/zimmer would be one nobody deploys, nobody reads, and nobody keeps true — the app never serves it, so nothing would ever surface the drift.

Nothing in Dockerfile is selective about this: the build stage does a blanket COPY . . and the final stage a COPY --from=build /rails /rails. The docs stay out because of a single /docs line in .dockerignore. That is a fragile place for an invariant to live — reorganize the file, or move the docs to another path, and the second copy comes back silently.

So two checks assert the outcome instead of trusting that line. Both run scripts/assert-docs-excluded.sh, which fails if it finds a top-level docs/ directory, or — anywhere under the tree it is pointed at — an astro.config.* or an @astrojs/starlight dependency in a package manifest (skipping node_modules, where a vendored Starlight belongs to somebody else’s dependency tree). A scan it could not run exits non-zero too: a guardrail that reports OK when its own machinery is broken looks exactly like a passing check, which is worse than no check at all.

The scan also skips the top-level tmp/ and log/ — anchored to the root it was pointed at, not matched by name, so a docs/tmp/ further down is still walked. Those are the two directories a running test suite scribbles scratch directories into, and a directory that vanishes between find’s readdir and its stat makes find exit non-zero, which reddens the guardrail over a race with whatever else is running rather than over the invariant. What the prune costs differs by caller: .dockerignore excludes /tmp/* and /log/* (keeping only the .keep placeholders), so for the build-context audit the prune skips ground that is provably empty, while against the built image /rails/tmp holds whatever the build’s own RUN steps left there — see the docs guardrail does not look in the image’s tmp/. For churn anywhere else, the two reasons a scan can fail are told apart by retrying it: a tree that changed underneath the walk clears on the next attempt, while a find that genuinely cannot run fails every attempt and still exits 2, saying how many it took.

Where	Against what	When it fires
`Dockerfile`, final stage	`/rails` — the published image’s own filesystem	during the release build, so an image carrying the docs is never pushed
`image_excludes_docs` in `ci.yml`	the real build context, via `Dockerfile.docs-audit` (busybox + `COPY . /ctx`)	on every PR, before merge

The second is a proxy for the first, and a tight one: COPY can only read from the build context, so docs absent from the context cannot reach the image. It exists because PR CI does not build the app image — pulling the multi-GB zimmer-base on a shared runner for a one-line assertion is not worth it — and catching the regression after merge is worse than catching it for a few megabytes of busybox. The gap between them (an ADD from a URL, a COPY --from an outside image, a RUN that fetches the docs over the network) is what the Dockerfile-side assertion is there to close.

Because the check is content-based rather than path-based, moving docs/ to a new directory without moving the .dockerignore line with it still fails. What it does not do is read .dockerignore and look for a line — a check like that passes happily while a COPY reintroduces the docs by another route.

Separately, release-image.yml carries paths-ignore: ["**/*.md", "docs/**"], so a docs-only push to main does not build an image at all.

Extensions do ship in the image

The mirror image of the rule above, and it is the one that regressed. An extension is a Ruby object that changes how Zimmer itself drives a runtime — which CLI adapter it spawns, which print-inference backend it uses, what environment it hands the child process. It only does any of that in the container that runs it.

.dockerignore used to carry /app/extensions/*/, and that one line made the whole seam unreachable in production. The failure was completely silent, by construction: ExtensionRegistry resolves builtins with safe_constantize and skips the ones that no longer resolve — the mechanism that makes deleting an extension safe — so the app booted, every seam fell back to native, and nothing anywhere reported that an extension had been asked for and not found. It stayed that way long enough for the only extension that ever shipped, mcp_tool_search, to be rewritten as a plain AppSetting column instead. That is #91.

The exclusion was there to demonstrate removability, and it conflated two different things. Removability is a property of the source tree: rm -rf app/extensions/<id>/ drops the feature, and the safe_constantize skip is what lets it drop with no core edit. Stripping the directory at image-build time proves nothing about that, and it guarantees that the one build meant to carry an internal-only feature never carries it.

The line is gone, and the opt-in mechanism it implied — scripts/install-extension.sh, a docker cp into a running container plus a restart — is deleted with it. It wanted a shell on the production host, which Ops actions ship with the deploy calls a defect to design out; and whatever it installed vanished at the next deploy. An extension merged to main is now in the next image, and an operator turns it on from Settings → Experimental. Nothing else. (One extension is not merged to main and never will be — see An extension can also arrive from outside the image.)

Enablement, not presence, is the safety property. A directory in app/extensions/ does nothing unless its class name is also in Zimmer::ExtensionRegistry::BUILTIN_EXTENSION_CLASSES — a core edit that goes through review — and Zimmer::Extension#default_enabled? is false, so even then it is off until somebody turns it on.

An absence is a hard thing to notice going missing, so two checks assert the outcome rather than the text of .dockerignore. Both run scripts/assert-extensions-shipped.sh, which fails if app/extensions/ is absent, if app/extensions/image_canary/IMAGE_CANARY.md is missing, or if any directory under app/extensions/ arrived empty. A scan it could not run exits 2 rather than reporting OK, for the same reason the docs guardrail does.

Where	Against what	When it fires
`Dockerfile`, final stage	`/rails` — the published image’s own filesystem	during the release build, so an image with the seam stripped out is never pushed
`image_includes_extensions` in `ci.yml`	the real build context, via `Dockerfile.extensions-audit` (busybox + `COPY . /ctx`)	on every PR, before merge

app/extensions/image_canary/ is not an extension: it holds no Ruby, so Zeitwerk’s collapse("app/extensions/*") loads nothing from it, and it registers nothing. It exists only so the check has something positive to find. The marker has to be one level down — the old rule excluded subdirectories and left app/extensions/CLAUDE.md sitting at the top of the tree throughout, so a marker there would have passed the whole time.

The empty-directory check is what covers the exclusions a single canary would not: a pattern like app/extensions/**/*.rb leaves every directory standing and hollows out the ones that carry code, and the canary, which holds no Ruby, would arrive intact. It is not depth-limited, because the code an exclusion strips is routinely one level further down than the extension directory — app/extensions/CLAUDE.md blesses a lib/ driver script inside an extension, and a scan capped at depth 1 would see pty_transport/ still holding lib/ and call the tree healthy.

test/infra/extensions_shipped_in_image_test.rb covers the half of this that needs no Docker daemon — that the detector detects (including the exact tree the old rule produced, so it cannot pass vacuously), that it does not fire on a healthy tree, that the canary is where the script looks, that .dockerignore carries no pattern naming the path, and that both callers are still wired up.

An extension can also arrive from outside the image

The rule above is about extensions whose code is in the repository. One is not: PtyTransportExtension (id pty_transport) depends on internal-only techniques that are deliberately not published, so BUILTIN_EXTENSION_CLASSES names the class and this repository carries none of its code. Production gets it as a read-only bind mount, from a host directory the private companion repo delivers alongside the catalog:

volumes:
  - /opt/zimmer/extensions/pty_transport:/rails/app/extensions/pty_transport:ro

Mount the extension’s own subdirectory, never /rails/app/extensions. Mounting the parent replaces the in-image tree with the host’s, which hides app/extensions/CLAUDE.md and app/extensions/image_canary/ — and the canary is the one thing the build guardrail above looks for, so the guardrail would go on passing (it runs at build time, against a tree no mount has touched) while the directory it protects was invisible at runtime.

A missing or empty host directory is inert: Docker creates the path, Zeitwerk’s collapse("app/extensions/*") finds no files, safe_constantize returns nil, the registry skips the name, and headless inference stays on native claude -p. So the mount is safe to declare on a host where nothing has been delivered yet — a fresh droplet, or a staging box that never gets the extension at all.

Production eager-loads, so the code has to be on disk when the container boots — a container already running when the files land does not pick them up. What reboots it is the delivery path everything else here uses: a health-gated kamal deploy cutover, which needs no shell on the box and holds the old container serving until the new one answers.

test/config/kamal_deploy_config_test.rb asserts the merged mount on both roles, and that nothing mounts over /rails/app/extensions itself. test/services/zimmer/extension_registry_test.rb asserts the other half: in this repository no built-in name resolves, so register_builtins! registers nothing and every seam is native.

Static files in `public/` are not digest stamped

config.public_file_server.headers in production.rb and staging.rb sets the cache header for everything under public/ — manifest.json, service-worker.js, 404.html, and the icons. It does not cover the compiled asset bundle: Propshaft serves that under /assets with its own far-future headers, and those filenames carry a content digest.

Nothing in public/ does. Each of those files lives at a URL that never changes, so a far-future max-age there pins whatever a browser fetched first — replace an icon or edit the manifest and the change reaches nobody who already loaded the old one. The header is therefore one hour, not one year.

An hour still leaves already-cached copies stale for an hour, and copies fetched under an older, longer header stale for as long as that header said. When you replace a file at a URL that has already shipped, bump its ?v= query in whatever references it — that is what the ?v=2 on /manifest.json and the two reused icon srcs is for.

The workflows

Every workflow but one runs on runs-on: self-hosted — a shared self-hosted runner pool that this repo registers against, so its CI stays off the GitHub-hosted Actions minute quota. If you fork Zimmer you would point these at your own runners (or switch the jobs back to ubuntu-latest). See Running on the shared self-hosted runner for what that requires of a Rails job.

The exception is alert-ci-failure.yml, which runs on ubuntu-latest on purpose: an alert that needs a healthy self-hosted runner in order to tell you the self-hosted runners are unhealthy is no alert at all. That buys less than it sounds like — it covers a degraded pool, where jobs run and fail, but not a pool that is flat offline, in which case runs simply queue (see CI failure alerts).

Workflow	Trigger	What it does
`ci.yml`	PR + push to main	rubocop · brakeman · `Gemfile.lock` freshness · `test-unit` (Postgres + Redis services) · `test-system` (Chrome browser suite) · `schema_verify` (round-trips `db/schema.rb` against `db/migrate/` on its own scratch Postgres) · GHCR-retention logic · docs site build · `shellcheck` over every tracked `*.sh` · `image_excludes_docs` (see The docs never ship in the image) · `image_includes_extensions` (see Extensions do ship in the image) · `all-checks-pass` (the aggregate gate). Every job except the gate is guarded to run only on `push` and on same-repo PRs, so a fork PR never checks out or executes fork code on the self-hosted runners. The gate itself is unguarded — it must never skip, or it would block branch protection — but it has no checkout step and only reads the other jobs’ results.
`pr-auto-close.yml`	outside PR opened/reopened	Zimmer does not accept pull requests: this politely comments and closes PRs from forks and non-members (owner/member/collaborator PRs are left open), pointing them at the issue tracker. Runs on GitHub-hosted `ubuntu-latest`, never the self-hosted pool.
`alert-ci-failure.yml`	any other workflow completing + manual	posts to #alerts in Slack when a workflow fails on `main`. See CI failure alerts
`release-image.yml`	push to main (ignores `*/.md`, `docs/**`)	rebuilds `zimmer-base:content-<key>` first when any of the five base inputs changed, then builds and pushes `zimmer:{version, latest, sha-…}`, retrying up to three times at three of the four points it touches GHCR — the login, the base manifest read, and the app build’s pull and push (the base build itself is still single-shot)
`build-base-image.yml`	manual + monthly cron	refreshes the base image for changes that move no repo input — base-OS/apt patches, the unpinned installers. Builds uncached to a scratch tag, verifies the published image against `Dockerfile.base`’s declarations, then (from main only) promotes it onto `zimmer-base:{content-…, latest}`. See Two paths rebuild the base image
`deploy-staging.yml`	manual only	see below
`teardown-staging.yml`	nightly cron + manual	`terraform destroy` of the staging droplet. The nightly run is conditional — it tears staging down only on the days nobody deployed to it (see Staging is torn down on the days it is not used). A manual dispatch destroys unconditionally; a powered-off droplet still bills, so destroying is the only way to stop the charge.
`ghcr-retention.yml`	weekly cron	prunes GHCR to ≤50 versions
`open-transcripts-drift.yml`	daily cron + manual + PR/push touching the vendored files	re-fetches the upstream OpenTranscripts files pinned in `vendor/open_transcripts/UPSTREAM.json` and fails when the bytes have moved (see Transcripts). Deliberately not on every PR — an upstream commit must not turn unrelated pull requests red. A scheduled failure reaches Slack through `alert-ci-failure.yml`.
`domain-cert-staging.yml`	weekly cron + manual + `workflow_call`	issues/renews the Let’s Encrypt cert for `var.domain` via ACME DNS-01 and pushes it to the droplet (see Custom-domain HTTPS). The scheduled run is guarded by the same droplet check as the teardown: with staging down there is no box to push a cert to, so it skips the week rather than failing.

CI failure alerts

When any workflow in this repo fails on main, alert-ci-failure.yml posts the repo, the workflow, the commit subject, the author and a link to the run into #alerts in the Tadasant Slack workspace. Any of your other repos can carry the identical listener under the identical secret names; if so, keep them symmetric and change them together.

It listens with workflows: ["*"], which matches every workflow in the repo — so a workflow added later is covered the day it lands, with nobody having to remember to wire it up.

It needs two repo secrets — SLACK_BOT_TOKEN and SLACK_ALERTS_CHANNEL_ID (Provisioning and secrets). Without them it logs a warning and exits 0: a missing alert secret must not turn into a second red X on main. With them, a Slack rejection does fail the job, because a rejected alert is a silently broken alert and this is the only place it can surface.

Three details worth knowing before you touch it:

It fires on an allowlist of conclusions — failure, startup_failure, timed_out — never on “not success”. ci.yml sets cancel-in-progress, so two pushes to main in quick succession cancel the first run, and a cancelled run must not page anyone. The corollary is that a run which never starts is never alerted on: if the self-hosted pool is offline, main-branch runs queue, and GitHub cancels them after ~24h as cancelled, which is indistinguishable from a deliberate cancel (Limitations).
["*"] matches the alert itself, and workflow_run chains several levels deep, so the job excludes itself by comparing against the literal name 'CI failure alert'. Rename the workflow and you must update that literal, or it starts alerting on its own runs (Limitations).
workflow_run only ever triggers from the copy of the file on the default branch, so editing it on a PR branch changes nothing until it merges. To prove Slack delivery works, run the workflow’s workflow_dispatch trigger by hand — it posts a smoke-test message instead of an alert.

Running on the shared self-hosted runner

The runner box is shared across several repos, so a job cannot assume it has the machine to itself. Five things follow. The first four are what every Rails job in ci.yml already does; the fifth is what every image-building job does:

ruby/setup-ruby gets self-hosted: true and bundler-cache: false. self-hosted: true selects the Ruby already staged in each runner’s own $RUNNER_TOOL_CACHE instead of downloading one — the action’s download path extracts into a hardcoded /opt/hostedtoolcache the runner user can’t write, so on this box the flag is mandatory, not optional. bundler-cache: false turns off the action’s automatic bundle install, because we do it ourselves into an isolated path (below).

The tadasant-zimmer-ci-* runners were registered with a Node toolcache but no Ruby one, so their Ruby 3.4.6 toolcache had to be seeded out of band (ruby-build 3.4.6 into /opt/hostedtoolcache-runner-N/Ruby/3.4.6/x64). That manual seed is not yet captured in IaC and will be lost if the runners are rebuilt — codifying it is part of the DigitalOcean migration (zimmer#118).
Gems install into a per-runner path. bundle config set --local path /home/runner/.bundles/zimmer-runner-${RUNNER_NUM} (with RUNNER_NUM derived from $RUNNER_NAME) keeps two concurrent jobs on the same box from fighting over one vendor/bundle.
Service containers publish dynamic ports. Postgres and Redis declare - 5432/tcp / - 6379/tcp (not 5432:5432), and a step resolves the assigned host port via ${{ job.services.postgres.ports[5432] }} into DATABASE_PORT / REDIS_URL. Fixed host ports would collide when two jobs land on the same runner.
The heavy suites are the test-unit and test-system job keys and pin PARALLEL_WORKERS. The runner’s file-based semaphore recognizes the job keys test-unit and test-system and caps how many heavy test jobs run at once; a bare test key would go ungated. Pinning PARALLEL_WORKERS stops a single job from fanning out to :number_of_processors (32 on this box) and starving co-tenants — test-system pins it to 1 because its persistent per-worker Chrome profile does not tolerate concurrent browser instances. test-system runs the Chrome-driven system suite (bin/rails test:system); the companion system-test semaphore gates it.
Image builds get a private DOCKER_CONFIG and name their builder explicitly. Every job that runs docker/build-push-action exports a fresh DOCKER_CONFIG under $RUNNER_TEMP before its first docker/* step, and passes builder: ${{ steps.buildx.outputs.name }} to each build. See Why image builds isolate their Docker config. test/config/image_build_workflows_test.rb fails the build if a workflow adds a build-push-action step without both.

Why image builds isolate their Docker config

All ~14 runner workers on the box execute as the same OS user with no per-job DOCKER_CONFIG, so they share one ~/.docker — one config.json and one buildx/current. Both are mutable state that any job can overwrite at any moment.

docker/setup-buildx-action creates a builder with docker buildx create --use, and --use writes the shared current-builder file. docker/build-push-action reads that same file to decide which builder to build on — it does not remember which builder its own job created. So a build step that starts after a co-tenant job’s --use lands silently builds on that job’s buildkit container. When the co-tenant finishes, setup-buildx-action’s post step runs docker buildx rm, which stops the container and deletes its instance file. The victim’s in-flight build sees:

ERROR: failed to build: failed to receive status: rpc error: code = Unavailable
desc = closing transport due to: ... received prior goaway: ... debug data: "graceful_stop"
ERROR: no builder "builder-<some-other-jobs-uuid>" found

The tell is that the UUID in the error is not the one the job’s own “Set up Buildx” step printed. The shared config.json has the matching hazard: a docker logout ghcr.io against it strips GHCR credentials out from under a concurrent job’s push. In build-base-image.yml that logout is docker/login-action’s post step; in release-image.yml it is an explicit final step, and it is guarded — see The login and the base resolve retry too.

A per-job DOCKER_CONFIG under $RUNNER_TEMP gives each job its own current-builder file and its own credential store, which removes both races; the explicit builder: input makes the binding unambiguous even if that state is ever shared again.

It is exported from a step rather than declared in job-level env::

- name: Isolate Docker client state for this job
  run: |
    cfg="${RUNNER_TEMP}/docker-config"
    rm -rf "$cfg"
    mkdir -p "$cfg"
    echo "DOCKER_CONFIG=$cfg" >> "$GITHUB_ENV"

The runner context is not available in jobs.<job_id>.env, so DOCKER_CONFIG: ${{ runner.temp }}/docker-config there expands to the empty string and silently points every build at /docker-config. A step’s run: always has $RUNNER_TEMP. image_build_workflows_test.rb asserts both the step form and that it precedes every docker/* step.

The release build retries GHCR, on the way in and on the way out

release-image.yml talks to GHCR at both ends of one step. It builds FROM ghcr.io/tadasant/zimmer-base:latest, so buildkit pulls that image’s layers for the length of the build, and it pushes the finished image at the end. When GitHub applies a secondary rate limit to the account — which it does across the whole account, not per workflow — either end starts failing.

Three shapes observed so far, one cause:

#10 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status from GET request to
https://ghcr.io/v2/tadasant/zimmer-base/blobs/sha256:…: 403 Forbidden
denied: permission_denied: … "You have exceeded a secondary rate limit."

ERROR: failed to solve: failed to copy: httpReadSeeker: failed open:
content at https://ghcr.io/v2/tadasant/zimmer-base/manifests/sha256:… not found: not found

#19 ERROR: failed to push ghcr.io/tadasant/zimmer:sha-…: unexpected status from HEAD request to
https://ghcr.io/v2/tadasant/zimmer/blobs/sha256:…: 403 Forbidden

The middle one is a lie worth recognizing: it reads as a missing manifest, but under throttling GHCR returns 404 for content it is simply refusing to serve — the same digest pulls fine minutes later. The third is the push side, and it is the reason the retry wraps the whole step rather than the pull: by the time it fires, the image is built and the only thing left to fail is the upload.

None of the three means the images are damaged or the credentials are wrong. On 2026-08-06 the same throttle took out this workflow and the production deploy in a different repo inside the same two minutes, and docker login had succeeded seconds before the deploy’s pull was refused.

Resolve base image is not what failed in any of them. It resolves the manifest through docker buildx imagetools inspect for real, and in the red runs it passed correctly — the image genuinely existed. What fails is the layer traffic after it, once the build is already underway.

Three attempts, escalating backoff, and a probe that says which side broke

The app build runs up to three times: Build and push, Build and push (retry) after 90 seconds, Build and push (final attempt) after a further 240 seconds. Every attempt but the last carries continue-on-error, and each is gated on all the attempts before it having failed. Only the last one can fail the job.

The backoff escalates rather than repeating because the throttle is account-wide and has outlasted a single 90-second wait. continue-on-error on the earlier attempts is load-bearing twice over: it swallows their failure, and it keeps the job green so that the implicit success() on the later attempts’ if lets them run at all. The two backoff steps between them carry it for the second reason only — a probe that exits non-zero would otherwise fail the job, and every later step, including the remaining attempts and the production notify, would skip on its own implicit success(). The retry chain would sit there intact and unreachable.

The retry is blind — it does not inspect the error. That is deliberate. The same throttle has already worn three different HTTP shapes, and gating on an error signature would trade a rare wasted rebuild for a missed retry the next time GitHub picks a fourth. Instead the gap between attempts runs .github/scripts/await-ghcr.sh, which reads a manifest from each package the build touches — the base image it pulls FROM and the app repository it pushes to, since a throttle need not hit both — and reports the result as a workflow annotation:

Either probe refused — GHCR was refusing this account, and the annotation says which package. The registry is the suspect.
Both answered — the build itself is the likelier suspect, so read the build log. Note this is evidence, not a verdict: both probes are reads, so they cannot clear a write-side throttle, which is the exact shape the 2026-08-06 push failure took. Check whether the build died on the pull or the push before concluding.

That is the line to look for on a run that exhausted its attempts, because the attempts themselves are unhelpful to read: a step that failed under continue-on-error renders with a red ✗ against a green job, since GitHub has no separate rendering for it.

The retry is not free, and the cost is lopsided in a useful direction. Every attempt runs on the same buildkit instance — builder: names the one this job created, and it lives for the whole job — so a retry resumes from that builder’s local cache rather than starting cold. A push-side failure is therefore cheap to retry: the image is already built, and the second attempt re-does little more than the export. A base-pull failure early in the graph is the expensive one, because there is nothing cached to resume from yet. (The GHA cache is no help either way on a retry: cache-to exports nothing from a failed build.)

The floor is the backoff itself. A genuinely broken build — one that fails for an ordinary reason and will fail three times — now takes 330 seconds of waiting plus three builds to go red, and concurrency: release-image with cancel-in-progress: false makes the next push queue behind all of it. That is the trade: slower bad news, in exchange for not paging anyone over a registry hiccup.

Every attempt takes its tag list from the tags output of Compute version rather than spelling it out three times, and image_build_workflows_test.rb asserts the chain stays wired: attempts in order, each gated on every prior one failing, identical build inputs, continue-on-error on all but the last, and a probing backoff step in every gap. Each of those, alone, is enough to produce a workflow that publishes nothing and reports the release green.

The build attempts are not the only place this job touches GHCR, and for a while they were the only place that survived it touching back.

Log in to GHCR was a plain docker/login-action step, which tries once. On 2026-09-02 that cost a release outright: 48 seconds in, ghcr.io failed to complete a TLS handshake, and every step after the login — buildx, the base resolve, all three build attempts, the production notify — skipped.

Error response from daemon: Get "https://ghcr.io/v2/": Get "https://ghcr.io/token?…":
net/http: TLS handshake timeout

Nothing about that commit caused it. The step now runs .github/scripts/ghcr-login.sh, which does the same docker login --password-stdin the action does — same registry, same github.actor, same GITHUB_TOKEN — up to three times, backing off 90 then 240 seconds, on the same reasoning and the same numbers as the build attempts. It is blind for the same reason too: this registry’s failures have worn a 403, two different 404s, and now a transport error with no HTTP status at all.

Two things it deliberately does not do. It does not retry an empty REGISTRY_PASSWORD — a missing credential is a configuration fault, and three attempts at it only delay the diagnosis by 330 seconds. And it does not carry continue-on-error: once the attempts are spent the login must fail the job, because an unauthenticated build fails later and less legibly. Registry output is fenced with ::stop-commands:: before being echoed, since a daemon error line beginning with :: would otherwise be read as a workflow command.

Because the login is a run: step rather than an action, there is no post step to log out. A final Log out of GHCR step with if: always() && env.DOCKER_CONFIG != '' replaces it. The logout itself is belt and braces — the credential lands in this job’s private $DOCKER_CONFIG and holds a GITHUB_TOKEN that expires with the job — but the guard is not. always() also fires on the runs where checkout or the isolation step itself failed, and DOCKER_CONFIG is empty on exactly those, so an unguarded logout would write to the shared ~/.docker above and strip a co-tenant job’s GHCR credentials. A post step could not reach that state, because it runs only if its own step did.

Resolve base image retries its imagetools inspect three times, 5 then 15 seconds apart. That read does not fail the job — it fails closed into need_base=true, which is worse in its own way: a hiccup escalates into a full base rebuild and push against a registry that may already be refusing the account. It cannot read the error to decide, because the one shape that would look like “genuinely absent” is a 404, and a 404 on a manifest is exactly what this throttle has already impersonated. The backoffs are seconds rather than 90/240 because the wait is paid on the ordinary path too: a base declaration that genuinely changed is absent, so it exhausts the retries every time. 20 seconds is a rounding error against the base build that follows; 5.5 minutes on every base bump would not be. When all three reads come back empty the step says so in a ::notice::, so a base build that then fails on registry errors reads as a throttle rather than as a broken Dockerfile.base.

Build & push base image is still single-shot. A throttle that survives the resolve’s retries and then kills the base build fails the job before the app build’s first attempt is ever reached. That path has not bitten yet; all the observed failures were the app build and the login.

build-base-image.yml also still logs in with a plain docker/login-action, and that is the deliberate half of the same line. It runs monthly and on workflow_dispatch, so a failed login is already in front of a human who can press re-run; release-image.yml is the one that fires unattended on every push to main, where nobody is watching and the next push queues behind it.

ghcr_login_test.rb drives the login script with docker stubbed on PATH and sleep stubbed out — succeed-first-time, fail-twice-then-succeed, never-succeed, empty token, and a forged ::error:: line from the registry — because a retry whose failure path has never run is not a retry, and ghcr.io cannot be asked for a TLS timeout on demand. image_build_workflows_test.rb holds the wiring: that the login runs the script rather than the single-shot action, that its backoff list is non-empty and numeric, that it runs after the DOCKER_CONFIG isolation and before the first build attempt, and that the base resolve’s read sits inside a loop with a backoff.

Staging deploys are Kamal container swaps onto a persistent droplet

The droplet is no longer cattle within a run of use. Terraform provisions it once and then leaves it alone; Kamal deploys the app onto it. (It is torn down between runs of use, on a nightly cron — see Staging is torn down on the days it is not used.) deploy-staging.yml:

Builds the base image (:staging) and app image (:staging-<sha>).
terraform apply — reconciles the existing droplet through remote state. It does not reap anything, and a re-run updates in place rather than recreating.
Joins the tailnet, resolves zimmer-staging’s peer IP from tailscale status --json, and loads the Kamal deploy key.
kamal accessory boot all -d staging, then kamal accessory reboot devdb -d staging, then kamal deploy -d staging --version=<tag> --skip-push. The boot line is unconditional, because kamal deploy on its own does not boot accessories and a newly declared one would otherwise never appear; it costs nothing to repeat, since accessory boot skips a host that already has the container. That skip is by existence, though, and a stopped container exists — so devdb, and only devdb, is rebooted rather than booted, making a deploy the way to recover a stopped one (#419). reboot stops and prunes the container before re-creating it, which is safe here only because devdb declares no volume; test/config/devdb_accessory_test.rb fails the build if that line ever names an accessory that does. Production’s pipeline, in the companion repo, runs both lines before its own deploy — accessory boot all and then the same single-accessory reboot — so a deploy is the recovery path on both destinations. Nothing else is: neither destination health-checks or alerts on that accessory, so recovery happens at deploy time and nowhere else (Limitations). Then the app rolls: kamal-proxy boots the new app container alongside the old one, health-checks it on /up, and only then flips traffic. A container that never goes healthy leaves the old one serving.
Re-verifies /up over the tailnet and asserts the worker container is running too — the worker is where agent sessions actually execute.
Smoke-tests the app it just deployed: /up/deep, a real page render, a CSRF round trip, and an Action Cable upgrade. See below — this is the step that decides whether the deploy is called healthy.

If the rebuild path (recreate_droplet) runs without TS_API_CLIENT_ID / TS_API_CLIENT_SECRET, scripts/tailnet-reap-node.sh still skips the stale-node cleanup — a fork that never configured a Tailscale OAuth client must not have its rebuild fail over it — but it now says so as a GitHub Actions warning rather than an info line nobody reads. The consequence of a silent skip surfaces much later and somewhere else: the rebuilt droplet registers as zimmer-staging-1 and the MagicDNS name drifts off the box you deployed.

Staging is torn down on the days it is not used

Deploy staging is manual, so between deploys the droplet just sat there. staging.tfvars.example sizes it s-2vcpu-4gb — $24/month at DigitalOcean list price, billed hourly — and staging was up for about thirty continuous days with deploys on a handful of them. A powered-off droplet still bills, so destroying it is the only way to stop the charge.

teardown-staging.yml therefore runs on a nightly cron (06:23 UTC) as well as on demand. The nightly run is conditional, and that is the whole design. The Kamal migration removed the old destroy-and-recreate-every-day churn for a good reason, and this does not bring it back: the cron tears staging down only on the nights nobody deployed to it. On a day staging is used, the run skips and the box stays warm for the next container swap.

scripts/staging-lifecycle-guard.sh is the decision, and both scheduled workflows ask it:

It checks	Because
Are there Spaces credentials?	A fork that never configured staging must not get a red X every night for a credential its owner never set — the same posture `tailnet-reap-node.sh` takes.
Is `digitalocean_droplet.zimmer` in the Terraform state?	State is what `terraform destroy` acts on, so “state manages no droplet” is exactly “destroy would do nothing”.
Did `Deploy staging` succeed within `RECENT_DEPLOY_HOURS`?	A day’s work keeps its box.

Both knobs live at the top of teardown-staging.yml, next to each other, because they are one decision: the cron says how often the question is asked, and RECENT_DEPLOY_HOURS (24) says how long a deploy protects the droplet for. At 24h against a daily run, staging outlives its last successful deploy by between one and two days. A deploy on any given day normally keeps the box through the next day’s run as well; the exception is one made in the ~20 minutes before the cron hour, which is already older than the window by the time that run fires.

A state with leftovers but no droplet is skipped, and says so. The guard looks for the droplet specifically, so a half-finished destroy — or a droplet deleted out of band — leaves resources the schedule will never clean up, including digitalocean_reserved_ip.zimmer, which DigitalOcean bills while it is unassigned. The guard prints them as GitHub Actions warnings rather than skipping silently; clearing them takes a manual Teardown staging dispatch.

A window shorter than the cron period is the trap. Set it to 18h and a run falls in the gap: on this repo’s real history that would have destroyed the droplet at 06:23 on 2026-08-16, thirty-four minutes before that morning’s deploy. test/config/staging_lifecycle_workflows_test.rb fails if the window drops below the interval between two runs.

Only the schedule is guarded. A workflow_dispatch skips the guard job entirely and destroys unconditionally — dispatching Teardown staging means tear staging down, including to clean up state after a droplet was removed some other way.

Why “nothing to do” has to be green

alert-ci-failure.yml pages #alerts on every main-branch workflow failure. A cron that reddened on the nights staging is already down would page nightly, forever, which is how an alert channel stops being read. So the guard exits 0 on every answer it can reach, including “no, skip”, and reserves a non-zero exit for the one case that genuinely needs a human: a Terraform backend it could not read at all. It never guesses “absent” from a broken backend — that would silently stop the teardown from ever running again, and look green while doing it.

The same bias runs through the deploy-history check. If the GitHub API will not answer — or answers 200 with a body that is not run history, which is what a truncated response looks like — the verdict is do not destroy: keeping a droplet nobody wanted costs about $0.80 for the day, and destroying one somebody is working on costs them a cold rebuild. Unknown is read as “in use”.

The one place the guard deliberately does not skip quietly is a repository that has staging and has lost the keys to its own remote state. A fork that never configured staging skips green — the posture tailnet-reap-node.sh already takes — but here that would switch the nightly teardown off for good, with the run still going green and the droplet billing round the clock again. STAGING_IS_CONFIGURED (set from whether a DigitalOcean token exists at all, a boolean about the secret and never the secret) is what tells the two apart without naming a repository.

domain-cert-staging.yml carries the same guard for the same reason. It upserts the staging.zimmer.tadasant.com -> tailnet IP A record and pushes the cert onto the box, so with staging routinely absent its own weekly cron would fail on a schedule of its own. Nothing is lost by skipping a week: deploy-staging chains that workflow after a fresh droplet comes up, so the cert is re-issued the moment there is somewhere to put it.

The `ref` input is resolved before anything is checked out

Deploy staging takes a ref — a branch, a tag, or a commit SHA — and pinning a deploy to a known-good commit is how a rollback is driven. actions/checkout only special-cases a full 40-character SHA, though; anything shorter it treats as a branch or tag name. An abbreviated SHA (9e95b4d) therefore had it fetch refs/heads/9e95b4d* and refs/tags/9e95b4d*, match nothing, retry three times, and fail sixty seconds in with

The process '/usr/bin/git' failed with exit code 1

which never mentions the ref. An abbreviated SHA is exactly what git log --oneline prints and what gets pasted into a pinned redeploy, and the input’s own description said “SHA”, so the trap was baited.

The job now checks itself out once to get scripts/ on disk, runs scripts/resolve-deploy-ref.sh, and checks out whatever that returns:

Empty — the exact commit the run was dispatched from (github.sha), so the two checkouts cannot land on different commits if someone pushes to the branch in between. No request made.
A full SHA — passed through untouched. Checkout already handles it, and asking the API about it could only narrow what the workflow accepts.
Anything else — one GET /repos/{owner}/{repo}/commits/{ref}, sent with the .sha media type so the answer is the forty characters and nothing else. That endpoint resolves branches, tags, and abbreviated SHAs alike.

Because the ref is interpolated into a URL path, it is first held to a deliberately narrow shape — [A-Za-z0-9._/@+-], never containing ... That is narrower than git check-ref-format: a branch legitimately named fix#123 is refused, and the message says so in those words rather than claiming the branch is invalid. Its full SHA is always a way through.

A ref that does not exist now fails in about a second, names itself, and quotes GitHub’s own sentence (No commit found for SHA: 9e95b4d). A ref that could not be resolved because the API was unreachable says that instead — with curl’s own reason attached — because those send an operator to two different places. So a 4xx is never retried, while a transport failure gets three attempts with a backoff between them.

`/up` is a liveness ping; `/up/deep` is the health check

/up is Rails’ built-in endpoint, and it answers 200 for any process that finished booting. A container with a dead database, an unreachable Redis, or a cache store that silently drops every write answers it 200 all the same. A deploy gate that asks only /up therefore declares a fully broken deploy healthy — which is what happened.

GET /up/deep (app/services/deep_health_check.rb) answers 200 only when every backing service the app cannot serve a page without has answered a real round trip, and 503 naming the one that did not:

Check	What it does	What only it catches
`database`	`SELECT 1`, then one indexed row from a real application table	A Postgres that connects but has none of the app’s tables — wrong database, unmigrated volume
`cache`	Writes a per-request canary and reads it back	`:redis_cache_store`’s `error_handler` swallows connection errors, so a dead Redis makes `write` and `read` return `nil` instead of raising. Reading the value back is the only way to tell a working store from one quietly discarding everything
`redis`	`PING` on the connection the cache store actually holds	Turns “the cache is not storing anything” into “Redis is unreachable, and here is what it said”. Reported as `skipped` where no Redis is configured (development, test), which is not a failure

{
  "status": "error",
  "failed": ["cache"],
  "checks": {
    "database": { "status": "ok", "adapter": "PostgreSQL" },
    "cache": { "status": "error", "error": "the cache store did not return the value it just wrote (read back nil)" },
    "redis": { "status": "error", "error": "Redis::CannotConnectError: Error connecting to redis://[redacted]@…" }
  },
  "checked_at": "2026-08-01T20:13:38Z"
}

Two deliberate properties. It is unauthenticated, like /up, so a deploy gate and an uptime monitor can reach it — which is why anything it echoes from a backing service is scrubbed of scheme://user:password@ credentials and truncated. And it is not behind HealthActionCooldown: that limiter guards the destructive maintenance actions and fails closed, so it would answer “rate limited” for precisely the broken-cache case this endpoint exists to report, and a health endpoint that refuses a monitor’s second poll in 30 seconds is not a health endpoint. What makes that safe is that the probe is cheap and fixed-cost — one SELECT 1, one single-row indexed read, one cache round trip, one PING, less work than any page the same visitor could request instead.

The post-deploy smoke step asserts four things, and every one of them fails the run (unlike the telemetry probe, which warns — broken telemetry is not a reason to withhold a working deploy):

Assertion	A failure means
`GET /up/deep` → 200	A backing service is down; the body names which
`GET /` → 200	The app is serving errors on its own root page
`POST` without a CSRF token → 422, then with the page’s token → 404	The session cookie or `secret_key_base` did not survive the deploy: every GET looks perfect while every form in the UI 422s. The target is a session id that cannot exist, so the authorized request 404s having changed nothing
…with web sign-in on	`GET /` is a 302 to `/login`, so the step renders `/login` instead (→ 200) and takes its CSRF token and cookie from there. The authorized `POST` then meets the login wall and answers 401 rather than 404. A 422 still means the token did not verify
`GET /cable` upgrades to a WebSocket	Turbo Streams cannot connect — every live update in the UI (timelines, status badges, notification counts) is dead

Two things follow from this that did not used to be true:

Rollback is one command. kamal rollback <version> -d staging (the host retains the last 5 images).
State survives a deploy. web and worker share durable named volumes (zimmer_data, claude_home, codex_home, pi_home, gh_config, claude_local), declared once in config/deploy.yml and re-attached to each new container instead of being destroyed with the droplet.

`CanaryJob` is what the post-deploy drain gate enqueues

/up/deep proves the web process can reach its backing services. It says nothing about whether the worker container is claiming jobs — and on 2026-08-13 a deploy passed every automated check while production processed zero background jobs for ten hours. The post-deploy drain gate closes that hole: it enqueues a canary onto all six scheduler lanes at negative priority. default, inference, pollers and triggers are gated; agents and auth are advisory because their threads can legitimately be held for an entire agent turn or interactive login.

The job it enqueues is app/jobs/canary_job.rb — a no-op that logs its token and returns. Everything about it is a constraint rather than a feature:

It touches no database, no network and no shell. Anything it starts touching is a way for a liveness gate to fail for a non-liveness reason, on every production cutover.
It declares no concurrency control. A total_limit/enqueue_limit rule makes GoodJob throw :abort at enqueue time, so no row is ever written; a perform_limit writes a row that is deferred rather than run. The gate cannot tell either from a dead queue — it would fail deploys of a healthy fleet.
It is not dead code, and its name is load-bearing. The gate resolves the class by name and falls back to a business job if it is absent. That fallback was CleanupExpiredElicitationsJob, which is a singleton sweep (include SingletonSweep → total_limit: 1) that runs every five minutes — so a canary enqueued onto it during a tick it was already running produces no row at all, and the gate reds a healthy deploy. Renaming or deleting CanaryJob reinstates exactly that.

One thing the gate still cannot see: queue recovery mode pauses default, inference, maintenance, pollers and triggers via GoodJob.pause, and that pause is persisted in good_job_settings, so it survives a deploy. A cutover that lands while recovery mode is active fails the drain check on all four gated queues with a perfectly healthy worker. The gate is the piece that has to learn to read GoodJob.paused(:queues) and skip rather than fail; nothing in this repo can do it for it.

test/jobs/canary_job_test.rb holds each of those lines, including a round trip through a real GoodJob row on all seven queues.

needrestart is told to leave sysbox alone, on every deploy

Keep needrestart from restarting sysbox (converge) runs scripts/install-needrestart-sysbox-dropin.sh against the staging droplet, before the sysbox preflight and before the cutover.

It exists because a host can wedge its own worker without anyone deploying anything. apt-daily-upgrade upgrades a library sysbox links against, needrestart restarts the sysbox units, and the daemons come back with an empty container registry — which orphans every sysbox container already running, permanently, while leaving their processes alive. Both hosts went that way on 2026-09-02, twelve minutes apart (#774). The step drops one snippet into /etc/needrestart/conf.d that declines the restart. The snippet is validated before it is published, not after: it is staged under a name needrestart’s *.conf glob cannot pick up, asserted against there, and only then moved into place — a file needrestart cannot parse aborts every upgrade run on the box, which is worse than the exposure it was meant to close. The assertions are that it compiles, that it deselects a real sysbox unit name while leaving a seeded stand-in entry intact (it does not read the host’s own needrestart.conf to check that), and that needrestart.conf is still somewhere the snippet gets read from. The mechanism and the one line it writes are in The host must not restart sysbox on its own.

Ordering is the point of putting it before the preflight: a droplet whose sysbox is broken enough to fail the preflight still gets the drop-in on the way past, and the worker container this run is about to create is covered from the moment it exists. Like the watchdog it runs unconditionally rather than gated on nested_docker, and it is a converge — the droplet is persistent, cloud-init runs only at first boot, and ignore_changes = [user_data] means a snippet written there would never reach a box that already exists.

Production is not on this path: its sysbox provisioning lives in the private companion repo and already converges the same override on every production deploy. Staging had no equivalent, which is what #775 was about.

The worker watchdog is converged on every deploy

Everything above proves the deploy is healthy at the moment it finishes. Install the worker watchdog (converge) installs the thing that keeps asking: scripts/install-worker-watchdog.sh drops scripts/worker-watchdog.sh on the host as /usr/local/sbin/zimmer-worker-watchdog and drives it from a 60-second systemd timer.

It exists because a container can pass every check in this document while running nothing. A cgroup OOM under sysbox-runc can leave the worker reporting Status=running, Restarts=0 with docker exec permanently broken (#502), so the probe is a real docker exec rather than a status read. What it does on a confirmed wedge, and the manual ladder for the rung it will not take on its own, are in When the worker wedges.

Two properties of the step itself. It runs unconditionally, not gated on nested_docker: an unexecable worker is worth catching under plain runc too, and a deploy that disarms sysbox should not silently disarm its watchdog. And it is a converge, not a one-off install — the droplet is persistent and cloud-init only ever runs at first boot, so re-running is how a changed script reaches a box that already exists. Same shape as Clear forced root-password expiry (converge).

Calling it from a deploy that is not this one

Production’s deploy lives in the private companion repo, and it does not reach its droplet the way the step above does: it goes over the tailnet with a generated ssh config, because its runner hygiene check forbids the key and the Host * stanza in the shared runner $HOME that a bare ssh root@host depends on. Two environment variables are the whole interface for that, and both are inert when unset — staging’s bash scripts/install-worker-watchdog.sh "$STAGING_HOST" above is unaffected by their existence.

Variable	What it does
`ZIMMER_WATCHDOG_SSH_EXTRA`	Extra arguments for every `ssh` the installer runs, split on whitespace. `-F <config>` is the motivating case. They go first, ahead of the installer’s own options, because ssh takes the first value it obtains for an option — so a caller can override `ConnectTimeout` or the host-key policy and cannot be overridden by them.
`ZIMMER_WATCHDOG_RECOVER`	`0` or `1`, rewritten into `/etc/default/zimmer-worker-watchdog` on every run. Unset — staging, and any host nobody has declared a value for — keeps the old behaviour: seed the commented template if the file is absent, then never touch it again, so an operator’s edit survives a deploy.

Recovery is the setting production cares about. It restarts the worker container, and on a host running real agent sessions that kills every one of them, so production runs detect-and-alert only. A value that merely starts right is not enough — it has to be re-asserted, including on a droplet rebuilt from scratch and on one somebody edited by hand:

ZIMMER_WATCHDOG_SSH_EXTRA="-F ${SSH_CONFIG}" \
  ZIMMER_WATCHDOG_RECOVER=0 \
  bash scripts/install-worker-watchdog.sh "$PROD_HOST"

What changed, and why

The old flow re-rendered the whole app stack into cloud-init’s user_data — a replace-forcing attribute on digitalocean_droplet. Any change to the image or an env var therefore destroyed and rebuilt the droplet (and everything on its disk). Combined with ephemeral Terraform state, which forced the workflow to hand-reap the droplet and firewall through the DigitalOcean API before every apply, staging was torn down and rebuilt constantly.

Now: the app stack lives in Kamal (config/deploy.*.yml), user_data is only a bootstrap, and the droplet carries lifecycle { ignore_changes = [user_data] } (deliberately not create_before_destroy — the tailnet hostname is fixed). A config or app change can no longer replace the box. The cost of freezing user_data is that the deploy key and Caddyfile can’t be updated in place — see Known limitations.

monitoring is in that same ignore_changes list, for the same reason from the other direction. The droplet asks for DigitalOcean’s metrics agent (monitoring = var.monitoring, defaulting to true) so a newly created box gets host CPU/memory/disk/load history for free, but the attribute is ForceNew in the provider — setting it on a droplet that already exists is a replace, not an update, and both environments apply with -auto-approve. Ignoring it keeps a routine apply from destroying the host over a metrics agent. The trade is that Terraform never gives the agent to an existing droplet, and DigitalOcean offers no API action to enable it, so an existing droplet has to get it from a deploy-time converge step. That step belongs in the production deploy, in the companion repository. Staging needs none, because no staging droplet in state predates monitoring and each new one is created with it set. See Known limitations.

ignore_changes = [user_data] now gates a second create-time-only feature the same way. var.node_exporter_enabled puts a tailnet-bound Prometheus node_exporter in cloud-init, so setting it produces no plan diff and installs nothing on a running box — a rebuild is what delivers it. See Known limitations.

Terraform, briefly

cd infra/terraform
cp staging.tfvars.example staging.tfvars
export TF_VAR_do_token=… TF_VAR_tailscale_auth_key=… TF_VAR_deploy_ssh_pubkey="$(cat ~/.ssh/kamal.pub)"
export AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…   # DO Spaces keys, for the state backend
# Operator keys, if you want the publickey door on :2222. In CI this comes from the
# ADMIN_SSH_PUBKEYS Actions variable; it is never committed. Unset means [].
export TF_VAR_admin_ssh_pubkeys='["ssh-ed25519 AAAA… you@laptop"]'
terraform init -input=false -backend-config=backend.staging.hcl
terraform apply -input=false -auto-approve -var-file=staging.tfvars

Creates: the droplet, the firewall, and a reserved IP (a stable public address across rebuilds). manage_project stays false by default — a project name is account-unique and a pre-existing one 409s, so a DO Project (just a console folder) isn’t worth the failure mode; flip it on with a one-time terraform import if you want one.

Terraform no longer knows anything about the app: no image ref, no secrets, no database wiring. Those are Kamal’s. It also does not create a DNS record — when var.domain is set, the domain-cert workflow owns the A record (pointing at the tailnet IP), which keeps the Cloudflare credential out of Terraform.

Staging runs a Postgres accessory container on the droplet, wired by Kamal — nothing external to provision. A self-hosted production deployment would instead point at its own database (Terraform can reference one as a read-only data source rather than creating it), but that lives in your own private infrastructure, not here.

→ Provisioning and secrets