Spawning and monitoring

app/jobs/agent_session_job.rb is the biggest file in the repo (~3,000 lines) and it is where Zimmer stops being a Rails app and starts being a process supervisor.

Its first decision is whether the turn may run at all. Every path that spends Claude quota — a first start, a follow-up, a fired wake trigger, a poller message, a restart — reaches this job, and for a spot session the job asks SpotSessionHold before doing anything else. A refused turn is deferred, not dropped: the job re-enqueues itself with the prompt and attachments intact and the session goes back to waiting. See Spot and priority. Only clone_only (no agent is spawned) and resume_monitoring (re-attaching to a process already running) skip the check, because neither spends anything.

What gets spawned

Claude Code:

claude --dangerously-skip-permissions \
  --disallowedTools Monitor ScheduleWakeup "Bash(sleep *)" "Skill(schedule)" AskUserQuestion \
  [--model MODEL] [--append-system-prompt SYSTEM_PROMPT] [--mcp-config PATH] \
  (--session-id UUID | --resume UUID) \
  -- <prompt>

Codex:

codex exec --json --dangerously-bypass-approvals-and-sandbox \
  --cd <working_dir> [-m MODEL] \
  --output-last-message <wd>/codex_last_message.txt [-i image]... \
  <prompt>

Both are spawned with pgroup: true (so the whole process group can be killed as a unit), stdin and stdout to /dev/null, and stderr to claude_stderr.log / codex_stderr.log inside the working directory — which is the clone root for a session without an agent root, and the agent root’s subdirectory for one with.

That distinction matters beyond spawn time. Everything that reconnects to a running process it did not spawn — a job resuming monitoring, ProcessLifecycleManager after a recovery spawn, the interrupt and terminate paths — has to rebuild this path, and both context-length recovery and failed-resume recovery are detected by reading the log. Rebuild it from the clone root, or with the wrong runtime’s filename, and those recoveries quietly stop firing. So there is exactly one way to ask: Session#stderr_log_path, which resolves the working directory and gets the filename from the session’s own adapter class (RuntimeCliAdapter.stderr_log_filename).

Why those tools are disallowed

Monitor, ScheduleWakeup, Bash(sleep *), and Skill(schedule) are all blocked because they are Claude Code’s own ways of waiting, and they don’t survive Zimmer. A background sleep loop dies when the container is recreated on deploy; a ScheduleWakeup doesn’t create a Zimmer trigger that Zimmer can track. Agents are pointed at Zimmer’s own MCP wake tools instead. AskUserQuestion is blocked because an interactive prompt would stall an autonomous session forever.

Runtime differences that leak

	Claude Code	Codex
Session ID	Zimmer generates it, passes --session-id	Codex mints its own; Zimmer captures it from the transcript
MCP config	--mcp-config <path>	~/.codex/config.toml (no flag)
System prompt	--append-system-prompt	Written into AGENTS.md below a marker
Resume	--resume UUID	codex exec resume UUID — and no --cd (the subcommand rejects it)
Transcript	plain .jsonl	zstd-compressed .jsonl.zst rollouts

The mints_own_session_id? flag on the transcript normalizer is what keeps these straight. Getting it wrong corrupts forked sessions — Claude’s session id must not be rewritten from the transcript, or a fork collides on the unique index.

What Zimmer appends to every prompt

AgentSessionJob#build_prompt_with_goal is the one prompt builder for both the initial spawn and every follow-up turn, so anything it appends rides along on every turn:

Block	When
The goal suffix	session.goal is set — a goal ID resolves to its description, free text passes through
<session-notes>	session_notes is non-blank
<session-hierarchy>	the session was spawned by another or has spawned one — see Hierarchy and human messages
<human-messages>	any session in that hierarchy has a human-authored message

The last two shrink to a pointer at the get_session_provenance MCP tool — the counts, and how to fetch the rest — when Settings → Experimental → Provenance context on demand is on, which is the default. Turning it off restores the full injected record. When each block appears at all is the same either way. See Hierarchy and human messages.

A blank base prompt is returned untouched, which is what lets the initial-spawn guard catch a task-less spawn instead of launching an agent on a bare goal string.

Large prompts and images switch transport

If images are attached, or the prompt exceeds LARGE_PROMPT_THRESHOLD (100 KB), the Claude adapter switches to stream-json mode and feeds the payload through an IO.pipe written on a background thread. A regular file doesn’t work here — the CLI reads nothing from it.

The pre-clone disk guard

Every session’s working directory is a git clone under ClonesDirectory.base, created by GitCloneService on the waiting → running path. CloneDiskGuard.ensure_space! runs immediately before the clone starts, and it does two things in order: it asks OrphanCloneFilesystemCleanupJob to reclaim space, and — only if that is not enough — it refuses the clone with a message naming the volume, the shortfall, and what to do about it. The refusal is a GitCloneService::InsufficientDiskSpaceError, a GitError subclass that is deliberately not classified transient: retrying a full disk on a five-second backoff accomplishes nothing, so AgentSessionJob fails the session and surfaces the message rather than rescheduling.

Without the guard, a clone into a full volume died partway with whatever errno git happened to surface, left a half-written directory behind, and — because that volume also holds every session’s scratch directory and prompt attachments — degraded every other session on the host at the same time.

How much space it asks for. A flat threshold fits this badly: a 50 MB repo and a 5 GB monorepo have very different needs. So the requirement is derived from the .git directory of the most recently written existing clone of the same repository, times SIZE_SAFETY_FACTOR (2 — one copy for the object store, one for the checked-out tree). .git specifically, not the whole tree: the tree also holds whatever the previous session installed (node_modules, vendor/bundle, build output), none of which the next git clone --single-branch will re-download, so sizing it would inflate the requirement by an amount that has nothing to do with the clone.

That measurement is bounded on four sides, because a sizing routine that errs pessimistically blocks every session on the host:

Bound	Value	Why
MINIMUM_FREE_BYTES	2 GiB, CLONE_MINIMUM_FREE_BYTES	Floor. A repo never cloned before, or one that cannot be measured, still has to clear it. Overridable so a small host has a lever that is not a redeploy
MAXIMUM_REQUIRED_BYTES	10 GiB	Absolute ceiling. A prior clone that grew pathologically must not become a requirement no healthy disk can satisfy
MAX_VOLUME_FRACTION	0.25	Relative ceiling. Without it the 2 GiB floor alone turns a 3 GiB disk that was cloning small repos perfectly well into one where nothing can launch
CLONE_SIZING_TIMEOUT_SECONDS	5s	The du runs on the launch path; exceeding the deadline falls back to the floor

The du is skipped entirely when free space already exceeds MAXIMUM_REQUIRED_BYTES — no requirement can ask for more than that, so a healthy host pays one df and nothing else.

It fails open. If free space cannot be determined at all — df missing, unparsable, or timing out — the guard permits the clone. A broken measurement must never be the reason no session can start; a clone that dies on ENOSPC is strictly better than that.

The spawn environment

Shared scrubbing (CliSpawnEnv):

• Loads a per-clone .env file if present (1 MB cap).

• Clears inherited env vars — DATABASE_*, RAILS_ENV, GEM_*, RUBY*, and a sweep of everything prefixed BUNDLE*. Without this the agent would inherit Zimmer’s own database credentials and Ruby toolchain. Four values are cleared for their own reasons: ZIMMER_OPERATOR_SSH_KEY (the agent gets the key’s path, not its material), ZIMMER_PARAMS_RESOLVER_SERVICE_ACCOUNT_KEY_JSON (the Parameter Store resolver credential — Zimmer resolves ${VAR} with it and injects the results a session’s MCP servers need, so the session itself never needs a key that reads every production secret value), and SENTRY_DSN_BACKEND (the production error DSN — an agent running bin/rails in a clone would otherwise report that clone’s exceptions as production errors), and ALERTS_ENABLED (the explicit opt-in that overrides AlertService’s environment gate — an instance that sets it to page must not hand that permission to every agent it spawns). A value in the clone’s .env always wins.

  This list is a denylist, not an allowlist. Sessions are plain child processes of the worker, so anything else in Zimmer’s environment is inherited verbatim — a new secret in env.secret is one env away from a transcript until it is named here.

• Sets AO_SESSION_SCRATCH_DIR — a durable per-session scratch directory. It lives on the zimmer_data volume, so it survives restarts and deploys, and it survives an archive/unarchive round trip intact. It is deleted when the session’s trash retention expires — see how long scratch lasts.

• Sets ELICITATION_REQUEST_URL and ELICITATION_SESSION_ID — where an MCP server sends an approval request, and who is asking. A value in the clone’s .env wins. This reaches the CLI, and on Claude Code the stdio MCP servers that inherit its environment; on both runtimes the stdio servers also get the two values from their own env table in the generated config, written by RuntimeConfigPostProcessor#inject_elicitation_env!. That second channel exists for the same reason as the SSH_PRIVATE_KEY_PATH forwarding below — Codex inherits neither — though the mechanism differs: a literal env table rather than Codex’s env_vars forwarding, so it also overrides a stale copy in a catalog entry’s own env.

• Sets SSH_PRIVATE_KEY_PATH — the operator SSH key the session authenticates with, when one is configured. The key file is written by OperatorSshKeyProvisioner; this exports its path, because an ssh-* MCP server looks for SSH_AUTH_SOCK and SSH_PRIVATE_KEY_PATH and nowhere else. A value in the clone’s .env wins. (Claude’s stdio MCP servers inherit the variable from the CLI; Codex’s do not, so the Codex post-processor forwards it explicitly through env_vars.)

Claude adds (ClaudeSpawnEnv): ENABLE_TOOL_SEARCH (see below), CLAUDE_CODE_DISABLE_CRON=1, CLAUDE_CODE_DISABLE_AUTO_MEMORY=1, CLAUDE_CODE_AUTO_COMPACT_WINDOW (default 1,000,000), and when MCP is on: MCP_TIMEOUT=180000, a clone-local NPM_CONFIG_CACHE, and one filesystem side effect — NpxBinExecutableGuard restores the execute bit on any bin target in that cache which lost it (MCP servers).

With session-scoped credentials on, Claude also sets CLAUDE_CONFIG_DIR — a durable per-session directory at ~/.zimmer/claude-config/<session_id>, alongside the scratch dir and reaped on the same schedule — and CLAUDE_CODE_OAUTH_TOKEN, the current account’s subscription access token. The child gets no refresh token, so it cannot rotate the subscription chain. Both are omitted when the setting is off, and omitted together when the pool has no current account holding a token, in which case the session reads the shared ~/.claude/.credentials.json as before.

Codex adds RUST_LOG=warn,rmcp=info and CODEX_HOME.

MCP tool search

ENABLE_TOOL_SEARCH is the one variable here an operator sets: it tracks the MCP tool search toggle at Settings → Experimental (AppSetting#mcp_tool_search_enabled), and it is on by default. On, Claude Code searches an attached MCP server’s tools on demand; off, it loads every attached server’s full tool schemas up front, which with several servers attached is a large, unavoidable context cost at the start of every session.

It is a Claude Code flag and nothing else reads it — CodexRuntimeAdapter never runs ClaudeSpawnEnv, so a Codex child never sees the variable at all, whatever the setting says.

Every session is tagged with what this setting was when it started and when it last ran, and the Costs page compares the two cohorts. See Experimental settings.

The setting is a plain column rather than a Zimmer Extension on purpose. It used to be the mcp_tool_search extension, which could not work in a deployed container: .dockerignore excludes /app/extensions/*/, so the class was absent from the image and the old ENABLE_TOOL_SEARCH=false baseline always stood in production. A column ships with the image. An enabled extension can still override the variable through the spawn-env seam below — extension contributions are merged last.

A spawn-env asymmetry

Zimmer::ExtensionRegistry.spawn_env_contributions is called only from ClaudeSpawnEnv — CodexRuntimeAdapter#spawn_process never consults it, so extension env contributions are unreachable for Codex, despite the hook receiving a runtime context that implies otherwise.

The elicitation variables used to be the other half of this pair. They now come from CliSpawnEnv, which both runtimes include.

The boot-tasks readiness gate

The last thing the job does before launching the CLI is check that the container it is running in has finished its background boot tasks.

bin/docker-entrypoint runs claude update and bin/ensure-playwright-browsers in a backgrounded block, deliberately: they are network-bound and can take 30s+, and running them in the foreground would hold Rails behind them until Kamal’s health check gave up. The cost is a window. ~/.local is a named volume that survives the deploy, so it shadows the CLI baked into the new image — until claude update finishes, the binary on disk is the previous deploy’s.

The entrypoint writes a marker file when that block completes, whatever the outcome, and exports its path as ZIMMER_BOOT_TASKS_MARKER. BootTasksReadiness.await blocks on the marker and the spawn path reports the result into the session’s log.

Outcome	What the session sees
Marker already present	Nothing. The overwhelmingly common case — the clone, air prepare, and MCP setup have already overlapped with the update.
Marker appeared after a wait	An info line naming the wait: Waited 4.2s for container boot tasks…
Marker says a task failed	A warning: the CLI may be the image’s version rather than the latest. Spawns anyway.
Marker never appeared	A warning naming the deadline, in the session log and in the process log. Spawns anyway.

Three properties are deliberate, because each of their opposites is worse than a stale CLI:

• Nothing here blocks Rails boot. This is read on the spawn path only. /up answers on schedule no matter what the background block is doing.
• The wait is bounded, and bounded from process start rather than from the call. If claude update hangs the marker never lands, and after ZIMMER_BOOT_TASKS_TIMEOUT_SECONDS (default 120) sessions spawn regardless. Because the deadline is anchored to when the process booted, the mechanism is inert once the container has been up longer than that — a session started an hour into a deploy never waits, and never can. A worker that refuses to spawn until a hung npm returns would be a worse outage than the bug.
• The gate is off unless the entrypoint armed it. Development, test, and bin/dev never run the entrypoint, so the variable is unset and await returns immediately without touching the filesystem.

The marker is per container, in /tmp, which is what you want: web and worker are separate containers running the same entrypoint, and each one gates on its own boot tasks. Sessions spawn in worker.

The monitor loop

Once spawned, the job loops: check the process is alive, poll the transcript file, broadcast new messages, repeat. Consecutive broadcasts are spaced apart, because a subscriber only sees SolidCable messages when its poll thread wakes: two broadcasts published inside one poll window can arrive coalesced, which is how a timeline renders messages out of order.

The spacing is derived from that window rather than restated: TranscriptPollerService.broadcast_spacing reads SolidCable.polling_interval — the same accessor the cable adapter itself sleeps on, parsed from config/cable.yml — and multiplies it by BROADCAST_SPACING_MARGIN (1.5). With the configured 100 ms interval that is the 150 ms this used to hardcode, but changing cable.yml now moves the spacing with it instead of silently breaking the relationship (#108). It remains a real throughput cost on a bursty transcript.

Two independent output channels:

• stderr → session logs. A thread tails the stderr file by byte offset every 0.5 s into a LogBuffer, flushed every 5 iterations.
• transcript → UI. TranscriptPollerService reads the JSONL, normalizes it, and pushes Turbo Streams. See Transcripts.

stdout is discarded for both runtimes, even though both CLIs are launched with a JSON streaming flag. The transcript file on disk is the only source of truth.

When the process exits

ProcessLifecycleManager#handle_exit asks the runtime’s retry strategy a series of questions, then — as a last recovery branch before giving up — checks for an abnormal signal death:

The diagram draws the recovery questions once, on the abnormal-exit branch, but handle_exit asks them on both: a normal-completion exit runs the same context-length, auth, API-error and failed-resume checks before it parks, which is how a failure that arrives with Claude’s exit 1 — session_id_conflict?, and the malformed tool call below — reaches them at all.

The auth branch does not re-inject blindly. AuthRecoveryCoordinator takes a per-runtime advisory lock on the account pool and picks one of three answers: adopt the account the pool already rotated to while this session was running (free — it is another session’s rotation), rotate away from the identity the runtime just rejected (re-injecting it would reproduce the wall), or wait for a rotation another process has in flight rather than starting a competing one. Which identity the session was spawned with is recorded in metadata["auth_identity_email"] at injection time; comparing it against the pool’s current account is what tells those apart. Decision tree in Agent harness auth.

When the login pool has nothing usable left — every account quota_exceeded, or an identity the runtime keeps rejecting — the session is parked rather than looped or failed: AuthOutageParkService explains the outage in the session log and the session-page banner, sends a push notification, and schedules a one-time wake-up trigger keyed off the real quota reset time. Creating that trigger sleeps the session, so it sits in waiting where the heartbeat sweep cannot nudge it. QuotaResetCheckerJob usually wakes it earlier, as soon as the accounts come back. Which of the two park reasons it gets follows the pool’s shape rather than the code path that arrived there — QUOTA_EXHAUSTED (“wait for reset”) when something is merely throttled, AUTH_UNRECOVERABLE (“re-authenticate”) when nothing is. Full detail in Agent harness auth.

A non-SIGTERM signaled exit — most commonly a cgroup OOM kill (SIGKILL) of a long-running, large-transcript session — is treated as recoverable rather than terminal: handle_signal_death resumes the existing runtime session id immediately (seconds, versus the ~15-minute stuck-session sweep), bounded by MAX_SIGNAL_DEATH_RETRIES so an OOM crash-loop can’t resume forever. The counter is reset once a resumed process runs stably, so a session that OOMs occasionally gets a fresh per-incident budget. Exhausting it fails with failure_reason: signal_death_retries_exhausted. AO-initiated SIGKILLs (the hung-process terminator escalating SIGTERM→SIGKILL) are excluded by the recovery_termination_initiated guard upstream of this check.

Failed resume is separate from process death. Claude reports it as a successful exit with “No conversation found”; Codex reports it as a failed exit with “no rollout found”. In both cases Zimmer starts a fresh runtime process against the same Zimmer session id. The prompt for that fresh start is chosen from the most durable in-flight source: active_follow_up_prompt, then sent_message, then pending_follow_up_prompt, then the original session.prompt. AgentSessionJob sets active_follow_up_prompt to the exact expanded runtime prompt for every follow-up turn before it clears the pending marker, including automated deploy continuations and status-summary forks that never had a pending marker. That slot is removed when the turn finishes normally — but on one path only, the :needs_input branch of the exit decision. The monitoring loop’s two fallback exits leave it set, so the slot is a reliable recovery source and an unreliable “this turn never ran” signal. AuthOutageParkService.park_undelivered_turn!, which guards those fallbacks, therefore checks the persisted transcript for the prompt rather than trusting the slot’s presence — see When the pool runs dry.

A turn that ends with the runtime having written nothing at all is the general backstop behind every specific branch above. A normal-looking exit over a completely empty transcript is not a completed turn — it is what “the agent never got going” looks like from the outside — so Zimmer restarts it from scratch instead of parking, bounded by MAX_EMPTY_TURN_RECOVERIES. “Nothing at all” is asked of both stores (RuntimeConversationPresence): Zimmer’s polled session.transcript and the runtime’s own file on disk, so a lagging poller can never be enough to abandon a real conversation. The invariant it restores: a failure Zimmer chose to retry never leaves the session at rest with an empty transcript and nothing driving it forward. Before it existed, a five-second npm hiccup during MCP connect could park a session in needs_input with a blank transcript until a human noticed and typed “continue”.

Two supporting rules make that reachable rather than theoretical. runtime_started is set the moment a pid is recorded, before the runtime has written a line — so when Zimmer kills a process that persisted no conversation, AgentSessionJob#terminate_process clears the flag, and the next turn spawns fresh instead of issuing a --resume that is dead on arrival. And when the runtime refuses a --session-id because that id is still held, Zimmer mints a new one and retries rather than reading the refusal — which Claude reports with its “turn complete” exit code 1 — as a finished turn.

Every replacement process is monitored. The recovery paths spawn through the same AgentProcessLiveness guard #spawn uses, and the job cleans up the lifecycle manager’s current pid rather than its own stale local copy — otherwise a replacement outlives the job that spawned it, stays on the clone, and keeps the runtime session id reserved.

When the job itself finishes, it reports the terminal status it actually reached. A job whose monitor loop already moved the session to failed closes its log at warning naming the failure_reason, exit_status, and exception_message it recorded — not Session job completed successfully. That success line was previously written unconditionally, so a Codex session killed by a failed resume ended its log claiming it had finished fine, which is precisely what a frozen session looks like from the outside.

Every one of those questions is answered by a regex against CLI prose

There is no structured exit signal. Zimmer determines why a session died by string-matching English:

• Quota exhaustion (which triggers account rotation): /hit your\b.*\blimit\b.*\bresets\b/i
• Auth loss: /not logged in|please run\s*\/login/i
• Context overflow: a pattern list

When Anthropic changed “hit your limit” to “hit your session limit” on 2026-06-14, account rotation silently stopped firing and the system retried six times against an already-capped account before giving up — with no log saying rotation should have happened. That outage is written up in the code. See Known limitations. Tracked in #53.

Falling off the end of that list is now loud. When none of the questions above answer yes, UnclassifiedFailureReporter logs and raises an #eng-alerts alert carrying the stderr tail and any transcript API-error text no pattern matched, before the session fails as it always did. It also fires on the inverse: a classifier that says a recovery path applies while that path’s service reports it does not. Dedup is keyed on the failure shape — runtime plus exit status, never the session — so a fleet-wide wave of one unknown mode is one page per hour, while a new shape pages on its own. A new failure mode that happens to share an already-seen runtime and exit code is suppressed for the rest of the window; the loud log is what catches that one.

Two scoping notes. This alert covers the failure branch only — a Claude exit 0 or 1 is a normal completion and never reaches it — but the normal-completion branch is not silent: it asks the same recovery questions and then the terminal-API-error backstop below, which pages on an unrecognized wording of its own. What lands in needs_input quietly is a turn that wrote real output and simply did not say anything Zimmer recognises as trouble. And a runtime whose strategy classifies nothing — Codex, until #3779 characterizes its transcript envelope — answers classifies_exits? => false and gets the loud log without the page, because for it “no classifier matched” is the designed-for path rather than news.

Not every “API error” in the transcript is the API

api_error_for_retry? reads the transcript’s isApiErrorMessage entries, and Claude Code writes one of those for a failure that never left the machine: a tool call the model emitted that will not parse. The CLI re-prompts the model in-turn — “Your tool call was malformed and could not be parsed. Please retry.” — and when that second attempt also fails it synthesises an assistant entry of its own (model: "<synthetic>", isApiErrorMessage: true, and no error field at all) and exits 1, its turn-finished convention.

That untyped entry is why ApiErrorRetryService::MALFORMED_TOOL_CALL_PATTERNS matches prose rather than an error type: there is no error type to read. It sits on the transient side because an unparseable tool call is a sampling artifact, not a permanent condition — and the CLI’s own in-turn retry does not settle that, since it re-prompts the same model with the same context, which is the worst conditions for escaping the failure mode. A respawn is a materially different draw.

What a retry cannot fix is a deterministically unserializable payload — an oversized tool argument that fails identically every time. MAX_RETRIES bounds that, and a ladder spent this way fails the session and pages #eng-alerts under “Malformed tool call survived every retry”, deduped per runtime. Deliberately louder than the generic exhausted ladder, which just fails with api_error_retries_exhausted: an exhausted 5xx ladder means the API was down for half an hour, but an exhausted malformed-tool-call ladder means Zimmer classified something as transient that isn’t.

Why this needed a classifier of its own

On 2026-08-25, production session 8878 finished an episode of real, paid work, downloaded the resulting MP3, and died on the upload — the last step — with

{"isApiErrorMessage":true,"message":{"model":"<synthetic>","content":[{"type":"text",
  "text":"The model's tool call could not be parsed (retry also failed)."}]}}

The entry carried no error type and no classifier owned the wording, so nothing retried it. handle_exit fell through to the terminal-API-error backstop, failed the session with failure_reason: terminal_api_error, and paged as an unknown failure mode. The alerting was correct — an unrecognized wording is supposed to be news — and the missing classifier behind it was the defect. One concrete instance of the general problem in #53; the fix is #668.

Metadata races

Session metadata and custom_metadata are JSON blobs that several processes write at once: the job’s monitoring loop, the web process, the GitHub pollers, and the transcript hooks. Writing one by rebuilding the whole column from a snapshot — update!(metadata: session.metadata.merge(...)) — erases any key another writer set since that snapshot was read. session.reload first narrows the window; it does not close it.

Some of those writers use Session#merge_metadata! / #remove_metadata! (and the custom_metadata equivalents) instead. Those push the merge into PostgreSQL as one statement — (metadata::jsonb - ARRAY[…]) || '{…}'::jsonb — so keys the caller never named survive.

session.merge_metadata!("process_pid" => pid, "runtime_started" => true)
session.merge_metadata!({ "process_pid" => pid }, [ "interrupt_terminate_pid" ]) # merge + remove
session.remove_metadata!(Session::SIGTERM_RETRY_METADATA_KEYS)

What that buys and what it doesn’t:

• Does: a write stops being destructive to keys it didn’t name. interrupt_terminate_pid (lose it and a “Send now” terminates nothing), pending_follow_up_prompt (lose it and a user’s message never reaches the agent), and github_pull_request_urls (lose it and no GitHub integration engages) survive that writer.
• Doesn’t: serialize two writers of the same key — last writer still wins. And atomicity is a property of every writer to the row, not of one key: a caller that still does a whole-column read-modify-write can erase a key no matter how carefully that key was written.

Most of the app is still that caller. Counted against this commit, app/ holds 34 atomic call sites across 15 files and 93 whole-column read-modify-writes across 27 files; AgentSessionJob alone has 23 of the latter against 11 of the former. The conversion is a long way from done — see Not every session metadata writer is atomic for which of the 93 are harmless and which are not, and #70 for the work itself.

The one worth knowing here: TranscriptPollerService batches metadata into the same update! as transcript and last_timeline_entry_at on every poll of a live turn, making it the worker’s single most frequent metadata writer. A key set in the window between its reload and its update! is still lost, so interrupt_terminate_pid is harder to lose than it was, not impossible. Splitting that batched write is what would close it, at the cost of a second write and an extra index broadcast on the hottest loop in the app.

Two deliberate differences from update!: model validations don’t run (which is what makes these usable on terminal paths, where a stale-catalog validation error would otherwise block a session from recording why it failed), and the after_update_commit broadcast callbacks are re-dispatched explicitly by the concern rather than fired by Active Record.

Stale job supersession

A session records the job driving its current turn in running_job_id, and the next job for that session has to decide what to do about it: stand down, so one turn runs at a time, or supersede it, because the worker that was running it is gone. Both wrong answers are silent. Respect a corpse and the user’s follow-up prompt disappears with no error anywhere; supersede a job that was merely slow and two agent processes run against one clone.

JobLiveness (app/services/job_liveness.rb) makes that call from evidence rather than from elapsed time, classifying the recorded good_jobs row as one of:

Status	Means	Next job
running	Locked by a GoodJob capsule that is demonstrably alive	Stands down
queued	Enqueued, not yet picked up — a worker will get to it	Stands down
scheduled	Parked on a future retry backoff (e.g. the transient-clone retry)	Stands down
dead_worker	Locked by a capsule that is gone: SIGKILL, OOM, evicted container	Supersedes
interrupted	Started, then lost its lock — the worker died mid-perform	Supersedes
abandoned	Sat queued and unclaimed past ABANDONED_QUEUED_JOB_AGE (30 min)	Supersedes

Liveness is asked of the database, not of the operating system. Zimmer runs the Kamal web and worker roles as separate containers with separate PID namespaces, and Kamal can spread roles across hosts, so Process.kill(0, pid) answers about the caller’s namespace: ESRCH for a healthy worker elsewhere, and “alive” for whatever unrelated process recycled the PID. GoodJob already keeps a registry every container can read — good_job_processes, refreshed on a 30-second heartbeat and, where GoodJob’s advisory_lock_heartbeat is enabled, pinned by a session-scoped Postgres advisory lock that dies with the worker’s connection. GoodJob::Process.active is the union of those two signals, and that is the probe. GoodJob’s default enables the lock in development only, so production and staging run on the heartbeat branch alone.

Existence is not liveness, so GoodJob::Process.exists? is the wrong question: a SIGKILLed worker leaves its row behind until some later capsule boots and runs GoodJob::Process.cleanup, so asking whether a row is there reports a dead worker as alive — which is how a follow-up prompt gets dropped.

ABANDONED_QUEUED_JOB_AGE is the one remaining clock, and it is a backstop rather than the mechanism: every ordinary death is caught by the two checks above, and this horizon exists so a job enqueued onto a queue no live capsule serves cannot wedge a session forever. It is deliberately far longer than any plausible queue delay, because crossing it early double-runs an agent.

Three callers ask this question, and they deliberately do not all ask the same one, because they are not deciding the same thing:

• AgentSessionJob reads the full status. It is deciding whether to run a rival agent right now, so it stands down on anything live, including a job that has merely been queued a long time.
• DeploymentRecoveryJob uses JobLiveness.alive?. It runs after a deploy has replaced the container holding the lock, which is exactly the case a locked_by_id-is-present test misses.
• CleanupOrphanedSessionsJob uses only JobLiveness.lock_holder_alive? and keeps its own 5-minute age gate. It is the periodic safety net whose whole purpose is to un-stick a session nobody is driving; deferring to ABANDONED_QUEUED_JOB_AGE would make it wait half an hour to do the one thing it exists for. Only the lock-holder question — where “the row exists” is a bad proxy for “the worker is alive” — is shared.

Two residual gaps, in opposite directions — how long a heartbeat-only deployment takes to notice a killed worker, and how a live worker can be mistaken for a dead one — are in Known limitations.

One live agent process per session

Superseding a job is not the same as ending the turn it was running. The agent CLI is a child of the worker process, and the ensure block that terminates it only runs if the job thread is alive to run it. A worker killed by SIGKILL or OOM, or a job thread killed at GoodJob’s shutdown timeout, leaves its agent process running and unsupervised. JobLiveness then correctly reports the job as dead_worker or interrupted — nothing is executing that row — the next job supersedes it, clones again and spawns. Two agents now hold one session: same feature branch, same $AO_SESSION_SCRATCH_DIR, same conversation resumed from a shared prefix, each believing it is alone. That is what #395 recorded, for sixteen minutes.

The fix is not to supersede less eagerly — that side of the decision drops prompts, and the table above is the best evidence available about a job. It is to ask a second, different question at the point of spawn: is the process the previous turn started still running? ProcessLifecycleManager#spawn is the single chokepoint every new turn passes through, and it calls AgentProcessLiveness (app/services/agent_process_liveness.rb) before launching anything.

That check is a PID check, which the section above rules out — for two reasons, both about a pid whose provenance was never recorded. So the provenance is recorded. Session#record_agent_process! writes process_pid and, in the same statement so they cannot drift, a process_identity holding:

• the kernel’s boot id (/proc/sys/kernel/random/boot_id), a random UUID regenerated on every boot of every machine;
• the PID namespace of the process that spawned it (/proc/self/ns/pid); and
• the process’s start time (field 22 of /proc/<pid>/stat), which distinguishes the process we started from any later process that inherits its number.

The boot id is not decoration. An nsfs inode number is unique only within one running kernel: pid:[4026531836] is the initial namespace on every Linux host, and the numbers restart after a reboot — when the start-time ticks have restarted from zero too. Namespace alone would compare equal across two hosts running the same role, and across a reboot of one. The three together mean “this kernel, this boot, this namespace, this process”.

Status	Means	At spawn
none	Nothing has been recorded for this session yet	Spawns
unknown	Recorded on another boot or in another PID namespace, or /proc is unavailable	Spawns, signals nothing
dead	Same kernel and namespace, process gone — or an exited-but-unreaped zombie	Spawns
recycled	Same kernel and namespace, number in use, but by a different process	Spawns, signals nothing
alive	Same kernel and namespace, present, and provably the process we spawned	Terminates it, then spawns

Only alive acts, and it terminates rather than refusing. The call carries the user’s prompt, so standing down here would trade a rare double-run for a silently dropped turn — the failure the whole supersede design exists to avoid. If the termination fails, that is logged and the spawn proceeds anyway.

It does not page. Two things reach that branch and they are not distinguishable at that point: a genuinely orphaned process, and a previous turn that was a second or two from exiting on its own when a fast worker picked up the next one. Terminating is right in both cases — by the time this runs, the previous turn is over — but alerting on it would be a false alarm most of the time. The record is a warning in the session log and a structured log line.

#spawn is the right place for it and #perform is not. Every new turn’s process comes through #spawn, and only new turns do: the monitoring-resume path deliberately reconnects to the recorded process and calls #resume_monitoring instead, so a check placed earlier in the job would terminate the very process that path exists to adopt. One case is knowingly swept up — an agent held alive across an MCP elicitation, which the monitoring loop keeps running on purpose so the in-flight tool call stays open, is terminated like any other, and that tool call is lost. A new turn is arriving either way, and two agents is the worse outcome.

This is the guarantee of last resort, not the first line. A job that is still running its monitoring loop ends its own turn when ownership moves — the loop reloads the session every iteration and terminates its process when running_job_id no longer names it — and an interrupt targets a specific pid through metadata["interrupt_terminate_pid"]. Both require the old job to still be alive. The spawn guard is what holds when it is not.

The same ownership question is asked one level down, in ProcessLifecycleManager#handle_exit. Several of its branches answer a process exit by spawning a replacement — the SIGTERM retry, the signal-death retry, compaction, the API-error retry — and each is right only while this job still owns the turn. Once running_job_id names another job, the exit being handled is very often one that job caused (the spawn guard terminating this turn’s process is exactly that), so a respawn would put a second agent back on the clone the guard just cleared. handle_exit stands down with :aborted instead.