Agent harness credentials

Zimmer keeps a pool of vendor accounts and rotates between them when one hits its rate limit. That’s the feature. Underneath it is the most fragile machinery in the project, and it’s fragile for a reason that isn’t Zimmer’s fault.

This is built on an undocumented, moving target

docs/CLAUDE_CODE_OAUTH_ASSUMPTIONS.md (now folded into this page and Known limitations) exists because Zimmer automates OAuth token management on top of Claude Code’s OAuth implementation, which is an undocumented internal. Every constant below (the token endpoint, the CLI’s client ID, the file paths, the on-disk JSON shape, the login prompt text) is a fact about someone else’s private implementation that can change without notice.

The assumptions doc was last verified against CLI 2.1.177 on 2026-06-14. Two production outages caused by exactly this are written up in the source.

The model

ClaudeAccount is the pool for the two runtimes that have one, discriminated by a runtime column (claude_code | codex). The naming is a leftover.

Everything goes through RuntimeAuthProvider.for(runtime) → ClaudeAuthProvider or CodexAuthProvider.

	Claude	Codex
Token endpoint	platform.claude.com/v1/oauth/token	auth.openai.com/oauth/token
Client ID	9d1c250a-e61b-44d9-88ed-5944d1962f5e (the CLI’s public ID)	app_EMoamEEZ73f0CkXaXp7hrann
Where a session gets its credential	CLAUDE_CODE_OAUTH_TOKEN in its environment, from the DB row; its own CLAUDE_CONFIG_DIR holds only mcpOAuth	~/.codex/auth.json, written before each spawn
Token TTL	from expiresAt (~8h, inferred)	24h, inferred — auth.json has no expiry field
Rotation	AccountRotationService, 5-minute interval	inline in the provider, 24h
Identity check on capture	email must match	none

Pi is not in the pool, and nothing here applies to it

Everything on this page — the pool, the refresh loop, per-session credentials, quota rotation, the drained-pool park, the login screen — is about a pooled subscription identity. Pi does not have one. It resolves a provider credential per request out of its own process environment, so there is no Pi row in claude_accounts, nothing to mark current, and nothing to rotate to. ClaudeAccount’s RUNTIMES inclusion validation would reject a pi row outright.

PiAuthProvider exists anyway, and every pooling method on it is a deliberate no-op rather than an unimplemented stub. It has to exist: RuntimeAuthProvider.for raises for an unregistered runtime, and SessionStatusSummaryGenerator, ProcessLifecycleManager, AuthOutageParkService and QuotaAvailabilityMonitor all pass session.agent_runtime straight into it. #accounts returns the always-empty pi-scoped relation rather than nil, because callers chain .available.exists? onto it and a relation is what makes that answer “no” instead of raising.

Pi is deliberately not in RuntimeAuthProvider::RUNTIMES. That constant drives the token-refresh dispatcher and the auth warm-up fan-out, and a runtime with no tokens to refresh has no business in either sweep.

What Pi needs instead is one environment variable, OPENROUTER_API_KEY, in the Parameter Store. A human sets it in the secrets console the Inference page’s Pi tab links; Zimmer does not write it. It lives in the store rather than in an account row, which is why setting it is a store write and not an account activation. PiRuntimeAdapter#apply_provider_key resolves it from the ${VAR} chain into the spawn environment at each spawn. See Runtimes and How the key reaches a Pi session.

Two consequences are worth carrying forward. #pool_exhausted? is permanently true for Pi, so a Pi session’s status summary is always produced by a headless completion rather than by forking the session. And a Pi 401 or 403 is terminal, not recoverable: PiRetryStrategy#auth_recovery_needed? is false by design, because the recovery on the other side of it is “rewrite the active account’s credentials and rotate to the next account” and Pi has neither. PiTurnError classifies such a turn :auth_terminal, which fails the session naming the provider’s own wording — a truthful end, where routing it here would park a human in front of a pool that does not exist. A Pi quota wall — a 402, or a quota-worded 429 — is not an auth failure: with no pool for AuthOutageParkService to wake on (PiAuthProvider#pools_accounts? is false), ProviderQuotaWallPark parks it on a timed re-check ladder instead; see Known limitations.

Session-scoped credentials: the DB owns the chain

Every Claude Code session spawns with its own CLAUDE_CONFIG_DIR and is handed a subscription access token through CLAUDE_CODE_OAUTH_TOKEN. It receives no refresh token, so it cannot rotate the chain — which is what makes “log in once, ever; Zimmer refreshes centrally” true rather than aspirational. There is no shared ~/.claude/.credentials.json in play: nothing in Zimmer reads it, nothing writes it, and the claude_accounts row is the only copy of the chain.

This is the only mode. It shipped behind an experimental setting in #626, ran that way in production, and the setting and the shared-file machinery it guarded were then removed. There is no toggle to turn it off.

Only Zimmer refreshes, under the pool lock it already had. Nothing else holds a refresh token, so there is no second writer to lose a race to — and that ownership applies to every part of a refresh. Zimmer imports nothing before a request, re-reads nothing after a rejection, and writes the rotated pair to nothing but the row.

The same rule reaches every claude process Zimmer starts, not only sessions. Print-mode inference — session titles, status summaries, push-notification copy — runs claude -p without a Zimmer session, so ClaudeHeadlessCredentials gives it the same two variables from the same row, keyed on one shared headless config directory (those invocations are stateless, so there is no conversation state to keep apart). Before that seam existed those calls inherited the worker’s environment, read the fossil shared file, and every one of them answered Not logged in · Please run /login and exited 1 — silently, because each caller falls back to a default. The only other claude invocation, the login flow, already ran in a scratch config dir.

Why this rather than more guards

~/.claude/.credentials.json had three writers and no owner. On 2026-08-22 Zimmer’s convergence write put a spent refresh token over the live one the CLI had rotated to; the CLI presented it, Anthropic answered invalid_grant, and the CLI blanked its own tokens. The credential then existed in neither store — data loss, not drift, which is why re-auth-and-wait never healed it. See #618.

Every mechanism that used to live on this page — the owner marker, the completeness guards on read and write, the symmetric write guard, sync_tokens_from_filesystem!, the corruption self-heal — existed to make that one file safe to share. Removing the sharing removed all of them, and none survives as a rollback: a file nothing writes is not a fallback, it is a fossil.

What the CLI actually does under it

Measured on CLI 2.1.240/2.1.241, against a scratch config dir on the production worker:

Question	Answer
Do OAuth MCP servers work with only an mcpOAuth block seeded?	Yes — connected and tools callable
Does the CLI write a claudeAiOauth block?	No. Top-level keys after a run: ["mcpOAuth"]
Where does a rotated MCP token land?	$CLAUDE_CONFIG_DIR/.credentials.json, mcpOAuth only
stdio and static-header servers?	Unaffected — the credential store is not in their path
Does --resume still work?	Yes, as long as the config dir is stable per session
Does ~/.claude/.credentials.json change?	No — byte-identical before and after

The file a session does write cannot destroy a subscription chain, because it never contains one.

Two behavioural differences worth knowing. The CLI writes no oauthAccount into the scratch .claude.json, so there is no filesystem identity to reconcile against — which is why the reconciliation surface could be deleted rather than fixed. And an env-var session gets user:inference scope only, which disables Claude-in-Chrome; Zimmer needs inference and MCP, both of which work.

The per-session directory

ClaudeSessionConfigDirectory resolves ~/.zimmer/claude-config/<session_id> — a sibling of the clones base, inside the same durable zimmer_data volume, so a deploy does not destroy it.

It is keyed on the session id and stable for that session’s whole life, which is load-bearing rather than tidy: Claude Code keeps its conversation state under CLAUDE_CONFIG_DIR, so claude --resume <id> only works when every invocation sees the same directory. “Fresh per session” means fresh per Zimmer session, not per process — a Zimmer session is a long-lived record resumed by many short CLI processes (p99 runtime 0.09h, max 1.27h over 42,971 runs, against an 8-hour token TTL: a token fixed at spawn has >6x headroom and needs no mid-process re-seeding).

projects/ inside it is a symlink to the shared ~/.claude/projects. CLAUDE_CONFIG_DIR relocates the transcript tree as well as the credentials, and Zimmer reads transcripts from the shared path in a dozen places. Credentials are what needs isolating; transcripts are not. The directory is reclaimed with the session’s scratch dir, and the cleanup deletes the symlink rather than following it.

MCP OAuth under it

ClaudeMcpCredentialWriter.for_session points the writer at the session’s own file — there is no host-global mode to point it at anything else, and .new requires the path. Rotated tokens are captured back by McpOauthRuntimeReconciler at every spawn and follow-up, unchanged — same server_name|sha256(…)[0,16] keys, same adoption rule, just a different path. One writer per file means the read-modify-write no longer races every other session on the worker; ClaudeCredentialStore’s lock still guards it, because a session’s spawn and its follow-ups can overlap the CLI’s own mid-session refresh of the same file.

RefreshMcpOauthTokensJob skips Claude Code. The cron has no session to scope to, and there is no single Claude file for a session-less sweep to read; ClaudeMcpCredentialWriter.session_scoped_store? is how it knows. The per-session reconciliation at injection time is what captures a rotated token instead — which leaves one narrow gap, named under Limitations. Revoking a credential through delete_runtime_credentials reaches the revoking session’s own store; a different session already running keeps its copy until it ends, and new sessions get fresh directories, so that window is one session’s lifetime.

When the pool cannot serve a token

The spawn fails closed. ClaudeSpawnEnv#apply_session_scoped_credentials raises MissingCredentialsError when no account is current or the current one holds no access token. The adapters let it through unwrapped and ProcessLifecycleManager refuses the spawn: the session fails with failure_reason: spawn_failed and a session-log line naming the account that came up empty, and the worker log carries it at .warn rather than .error — an empty pool is a configuration state that is already red on /health’s Agent Authentication card and already alerted through the account_needs_reauth event when an account died, so paging once more per spawn attempt would be noise.

It used to fall back to the shared file, and that was right while the file was a live rollback. Falling back now would point the child at a fossil and every turn would come back Not logged in · Please run /login with nothing in the log to say why. A session that cannot authenticate is a session that cannot work, and failing its spawn is the honest outcome. It is a failed session rather than a parked one, which differs from how a pool that drains mid-session is handled (When the pool runs dry parks into waiting): the park machinery is built around a turn that reached the runtime, and this happens before there is a process. AuthWarmupService settles a usable current account at worker boot so the ordinary deploy never reaches this.

There is no rollback, and what recovery looks like instead

Unticking a box no longer restores the shared-file behaviour, because there is no box and no machinery behind it. Recovery is the same one gesture in every case: Authenticate the account from /inference. That drives an interactive login in a scratch directory, validates the pair Anthropic hands back, and writes the row — and the next session to spawn reads the new token out of it. There is no second store for the write to fail to reach and no separate Switch step to make it take. If the current account is the broken one, Authenticate it; if another account is healthy, Switch to it and every spawn from then on runs as that account.

The operator-facing reconciliation surface went with the second store. The “Filesystem identity mismatch” banner, the “Sync from filesystem” button and its route, ClaudeAccount.sync_from_filesystem!, ClaudeAccount.filesystem_oauth_email, Claude’s reconcile_filesystem_identity!, the filesystem auto-capture that bin/rails claude_accounts:add used to perform, and the claude_accounts:capture_tokens rake task are all gone. Asking an operator to adjudicate between two stores was never the right answer to a disagreement, and the banner’s own copy told them to run a bin/rails command on the worker — which production invariant 11 forbids. A Claude account’s card on /inference has two verbs, Authenticate and Switch, and nothing that asks anyone to reconcile, adopt or sync. (A Codex card keeps Re-activate for the current account, because ~/.codex/auth.json is a real file that can need rewriting.)

Deleting an account keeps its history

Delete is a real delete: the claude_accounts row goes, and the account leaves the pool. What does not go is the record of how it behaved. Its quota snapshots, its login attempts, and every rotation event pointing at it are detached (dependent: :nullify) rather than destroyed.

That is not tidiness, it is a diagnosis problem. The operator response to a misbehaving account is “delete it and re-authenticate” — the two buttons sit side by side on every /inference card — and the cascade used to take the evidence with it. On 2026-07-31 an account was deleted and re-added, and the replacement row read Quota snapshots: None plus one login attempt: indistinguishable from an account that had never once completed a call. The row was thirty minutes old and said nothing about the credential at all.

:restrict_with_error would have preserved the same history by refusing the delete, and was rejected on those grounds: any account old enough to be worth diagnosing has history, so Delete would become a control that only ever errors.

A detached row still has to name who it was for, or it preserves bytes rather than evidence. Each carries the identity at write time:

Table	Carries	Why
claude_account_quota_snapshots	account_email, account_runtime	a reading attributable to nobody answers nothing
runtime_login_attempts	account_email	”did this credential ever log in?” outlives the account, within the 1-day attempt retention window CleanupRuntimeLoginAttemptsJob already enforced
account_rotation_events	rotated_from_email, rotated_to_email, runtime	the pool moved from and to something, and /inference filters the table by runtime

The rotation table’s runtime column is load-bearing. /inference used to scope rotation history by joining to the target account’s runtime, which an event whose target has been deleted cannot satisfy — so preserving the event without it would have dropped the event off the page it exists to inform. The preserved emails also restore a distinction the old :nullify on rotated_from_id erased: a nil source with no email is a rotation that genuinely had no source (a bootstrap), and a nil source with an email is one whose source was deleted. The page labels the second kind “deleted”.

Two consequences worth knowing:

• Re-adding the same email creates a new account row that inherits none of the old one’s history. The history is still there, attached to the email rather than to the new id — readable in the rotation log and in /supervisor, not on the new card.
• claude_accounts:clear_all / codex_accounts:clear_all still destroy everything, detached rows included — they find those by the denormalized runtime, since there is no foreign key left to find them by. They are the deliberate start-over affordance, and they say so.
• Nothing else removes a detached quota snapshot. claude_account_quota_snapshots has no prune job, so an account’s readings are now permanent unless clear_all takes them. See limitations.

The refresh loop

RefreshRuntimeAuthTokensJob runs every 5 minutes:

The refresh threshold is 15 minutes on a 5-minute cron — three chances to catch a token before it expires.

Refresh tokens are single-use and rotating

Every refresh returns a new refresh token and invalidates the old one — and invalidates the sibling access token. Two consequences the code has to defend against:

1. The new pair must be persisted atomically. A crash between “got new tokens” and “wrote them” bricks the account. refresh_token! writes both in one update!.
2. A future expiresAt is not proof a token is live. If someone else refreshed, your still-unexpired access token is already dead. The code does not enforce this — token_expired? still keys purely off expiresAt. The defense is the completeness invariant.

A credential set with no refresh token is a dead end. ClaudeAccount.complete_claude_oauth? gates the two paths that put a Claude credential into the row — an interactive login’s capture, and the health read that decides whether the pool can serve a session at all — so an incomplete set never enters the pool. It used to gate a filesystem read and a filesystem write as well; those paths are gone.

Whose tokens are in play

The is_current row’s. That is the whole answer now, and it is why the question stopped being interesting: there is no on-disk identity file to disagree with the DB, no shared credentials file whose owner has to be recorded in a sidecar marker, and no container-local ~/.claude.json whose different durability could make a reader confidently wrong. /health’s Agent Authentication card names the current account under Session credentials (database).

Credential health

ClaudeCredentialHealth classifies the credential a Claude session would be spawned with as :ok, :absent or :corrupt. :absent — no account is current yet — is what a fresh deployment looks like before its first login, and the next spawn selects one. :corrupt — an account is current but its stored pair is unusable — means every session spawned from it is logged out, which is why it is the same state a blanked shared file used to be rather than a new one.

It has a surface and no repair, and both are deliberate. The surface is the Agent Authentication card on /health, critical while the row is corrupt and folded into the dashboard’s overall status; CliStatusService reads the same classification for the Claude Code tile on the CLI status page, because no claude invocation can answer “is the stored credential usable”. There is no self_heal! because a corrupt file could be rewritten from the DB, but a corrupt row is the bottom of the stack: only a human re-authenticating from /inference fixes it, and the card says so rather than promising a repair that cannot work.

Rotation on quota

When an account hits its rate limit, Zimmer rotates to the next one by priority:

1. Snapshot the outgoing account’s quota state.
2. Label the outgoing account on evidence — see A rotation is not evidence about quota.
3. On a quota rotation, re-rank every account’s priority by its ClaudeAccountScore, best first (see Proactive rebalancing), so the accounts with the most weekly usage sink to the bottom before a successor is chosen. An account with a spent window that has not reset ranks last whatever its score.
4. activate_next_account — which skips a candidate whose latest snapshot says its weekly window is spent, then validates the survivor by calling refresh_token! before activating it. A broken or capped account is skipped before it can be handed to a session.
5. Mark the new account current. Nothing is written to disk — the next spawn reads its token out of the row.
6. Record an AccountRotationEvent.

Steps 1–6 run under the per-runtime pool lock (ClaudeAccount.with_pool_lock), and the caller passes the identity its session was running as. A stampede — N sessions hitting the same account’s quota within seconds — therefore produces one rotation: the first racer moves the pool, and every racer behind it finds the pool already off the account it was complaining about and returns that new account instead of rotating again.

Without that, a stampede drained the pool

Anthropic’s refresh tokens are single-use. Unserialized, N racers read the same current, selected the same successor, and each called refresh_token! on it; the first rotated the token and the rest got invalid_grant, which was read as permanently dead and marked needs_reauth. Four different accounts died that way in ten days, and account_rotation_events carries the fingerprint — duplicate and triplicate from → to pairs seconds apart. Serializing stops the concurrent refresh; collapsing stops the serialized racers from each burning one more account. See #242.

Proactive rebalancing

Rotation on quota only leaves an account once it is exhausted, so on its own it would keep the pool on an account at 81% of its week while untouched accounts idle. AccountRebalanceJob runs every five minutes and moves an idle pool onto its best account before anyone needs it.

Every account is scored by ClaudeAccountScore, a number from 0 to 1:

score            = 0.8 × headroom(7d) + 0.2 × headroom(5h)
headroom(window) = (1 − u) + u × relief
relief           = clamp(1 − time_to_reset / horizon, 0, 1)    # horizon: 24h for 7d, 5h for 5h

u is the window’s effective utilization. The weekly window carries most of the weight because a point of it is gone for days. Usage that will come back soon counts partly as headroom, so an account whose week resets in six hours is worth spending down and one with a day and a half left is not.

The job switches only when all of these hold, and AccountRebalanceService.plan reports the first one that does not:

Condition	Why
no turn is on a worker or queued for one (Session.running_turns, any runtime)	a switch must not land between two turns of a live session
no Claude Code session has shown transcript activity for 60 minutes (CLAUDE_REBALANCE_IDLE_MINUTES)	Anthropic’s prompt cache is per account and lives an hour; after that a switch throws nothing away
nobody pressed Switch on /inference in the last 24 hours	a human’s pick had a reason the score cannot see
the current account has a reading less than two hours old	the comparison needs two numbers
another account is eligible: active, credentialed, read in the last two hours, under 90% of both windows	anything else could wall the next busy stretch mid-turn
the best one beats the current account by at least 0.15 (CLAUDE_REBALANCE_MIN_SCORE_GAIN)	hysteresis, so near-equal accounts do not swap every hour

A current account that cannot serve at all (not active, or its credential refused) scores 0 and skips the manual hold and the freshness check. A fleet whose running turns cannot be read holds.

CLAUDE_REBALANCE_ENABLED=false turns it off. The switch holds the pool lock, probes the candidate once (which also proves its credential), re-checks for a turn in flight, and records an AccountRotationEvent with reason rebalance. It never marks the outgoing account: a rebalance says nothing about its quota.

A starting session never waits on a rebalance. mark_current! flips is_current in one transaction, so a spawn reading current_account sees the old account or the new one, both valid, and never neither. A session that reads the old one just before the commit runs on it; the next spawn gets the new one.

Each quota rotation and each rebalance re-ranks priority by the same score, so the reactive path falls back to the account the proactive path would have chosen. A priority typed into the add-account form is therefore only a starting position.

/inference shows the decision under Accounts: a sentence naming the condition that is holding the switch, or the switch that is due, and every account’s score.

Codex is not rebalanced and its priorities are not re-ranked: a Codex account has no periodic quota reading to score. Pi has no account pool.

A rotation is not evidence about quota

Step 3 used to be an unconditional mark_quota_exceeded! on whatever account the rotation was leaving, whatever it was leaving for. For a rotation on auth_recovery that label is a fabrication: the runtime said “Not logged in”, which says nothing about quota — and because status is what ClaudeAccount.available reads, fabricating it does not merely mislabel one account, it removes it from the pool.

So the outgoing account is labelled only when something observed says so:

Observed	Outcome
already needs_reauth, or already quota_exceeded	left alone — the two statuses drive different recoveries, and marking twice counts one wall as two quota hits on /inference
the reading step 2 just took says either window is spent	quota_exceeded, whatever the rotation was for — a live reading is the strongest evidence there is
the caller rotated for a quota wall (reason: "quota_exceeded")	quota_exceeded — it watched the runtime refuse the request, which is evidence even when the probe could not be taken
anything else	left active

A reason the list does not recognise falls in the last row. Over-labelling is the failure this rule exists to prevent, so a new rotation reason has to opt in rather than be assumed in.

The reading is asked as !windows_clear? — the same predicate effective_status renders and QuotaResetCheckerJob restores on — so a label rotation writes is one the rest of the app honours. The narrower five_hour_window_spent? would write labels windows_clear? immediately overrules: a mark nothing acts on, over an account every spawn path still refuses.

CodexAuthProvider#rotate_under_lock applies the same reason gate, and it is the only gate it can apply: a Codex account carries no Anthropic quota window to probe, so there is no reading to weigh. That also makes the mistake permanent on that side rather than merely slow — QuotaResetCheckerJob restores a Codex account only on the rate-limit reading Codex recorded when it refused the account (see Codex quota and auth), and a label written by mistake has none.

Without that, one blanked credential read as a drained pool

At 02:05Z on 2026-08-23 a Claude account’s OAuth tokens were blanked to empty strings, so every session on the worker was logged out at once. Each one rotated away from the account it was holding; each rotation stamped quota_exceeded on the account it left; in about forty seconds every row in the pool wore the label with no quota reading behind any of it. activate_next_account then ran out of candidates, park_reason_for_pool read the labels, and four sessions were parked with “Quota exceeded across all Claude Code accounts” against a noon reset estimate — ten hours out.

The proof the labels were invented: QuotaResetCheckerJob restored all three accounts minutes later, on their own readings, and it only restores an account whose snapshot is windows_clear?. The /inference badge had been working around these labels since #426; the parking decision was still reading them raw.

Codex quota and auth

A Codex session reaches the same rotation and recovery paths through the error Codex records on a failed turn (see Agent harness):

• usage_limit_exceeded — “You’ve hit your usage limit … try again at 2:23 PM”, and also “Quota exceeded” and “To use Codex with your ChatGPT plan, upgrade to Plus” — goes to ApiErrorRetryService, which answers :quota_exceeded without retrying, so ProcessLifecycleManager rotates through CodexAuthProvider#rotate_for_quota! and resumes the thread on the next account. With nothing to rotate into, the session parks with QUOTA_EXHAUSTED, exactly as a Claude session does.
• The reading that says when the account is back. Codex records the account’s rate-limit windows — used_percent, window_minutes, resets_at from the backend’s x-codex-primary-* and x-codex-secondary-* headers — on the token_count record it writes just before the refusal. ApiErrorRetryService keeps that as a usage_limit quota snapshot against the account the process ran as (primary window in the five-hour columns, secondary in the weekly ones), and QuotaResetCheckerJob restores the account once windows_clear? says its capped windows have reset. The capped windows are marked rejected; a refusal the reading does not explain (no window at its cap, or a capped window with no reset time, or no windows at all) is written as a refusal with no reset, which never reads as clear. Every refusal writes one, so the newest snapshot — the one the sweep and the /inference heal both read — always describes the newest refusal, and the same reading gives a Codex quota park its “resumes at” estimate.
• unauthorized — “Your access token could not be refreshed because your refresh token has expired / was revoked / was already used” — and a 401 go to AuthRecoveryCoordinator. When the pool has not moved under the session, the coordinator refreshes the account first. For a Codex OAuth account a refresh that succeeds proves the account serves (CodexAuthProvider#refresh_proves_serviceable?: Codex reports quota with its own code, so this one is only about the credential), so the session is re-seeded with it instead of being rotated away from a working account — the usual cause is a Codex process losing the single-use-refresh-token race to another refresher. That happens once per incident; failing the same way again rotates. A refresh that fails condemns or strikes the account as usual and rotates, and an API-key account — whose “refresh” is a no-op that proves nothing — rotates straight away.

A Codex account marked quota_exceeded by a refusal that left no reading stays marked until someone re-activates it — see Known limitations.

Refreshing a token without burning it

Both vendors issue single-use refresh tokens: a successful refresh returns a new pair and invalidates the old one. Present a spent one and Anthropic answers invalid_grant, OpenAI refresh_token_reused — and neither response can tell you whether the credential is dead or whether somebody else simply got there first.

ClaudeAccount#refresh_token! has nine call sites (the Inference page ×4, QuotaResetCheckerJob ×2, rotation, activation, and the 5-minute RefreshRuntimeAuthTokensJob sweep), so both protections live in that one method rather than at each of them:

1. A row lock across the whole read-refresh-persist sequence — the only scope that can promise the token being presented is the token still held. On acquiring it the method re-reads the row: if another caller rotated the token while this one queued, their refresh is this one’s refresh, so it reports success rather than spending a token nobody has used yet.
2. A lost-race check before condemning anything. The lock excludes other Zimmer callers but not the agent CLI, which rotates the shared credential file mid-session. So on a permanent-looking failure the method re-syncs from disk and compares: if the token of record has moved on from the one it presented, this was a lost race, and the account stays active instead of going needs_reauth.
3. Strikes, for the failures that check cannot see. The disk holds one account’s credentials at a time, so the lost-race check has evidence only for the account that owns them — for the other five in the pool it answers “not a race” because it has nothing to compare, not because nothing raced. Condemning on that answer condemns on nothing, so it no longer does: see below.

Both are runtime-agnostic — the Codex path has the same single-use semantics and gets the same protections. API-key Codex accounts skip the lock entirely: nothing to rotate, no race to lose.

A rejected value is not a dead credential

An invalid_grant means one of two unrelated things, and the HTTP status cannot tell them apart. The description can:

Vendor says	Means	Zimmer does
invalid_grant + Refresh token expired (or revoked), 401, 404, invalid_client, unauthorized_client	the credential is finished	mark needs_reauth at once — unless the lost-race check can prove the row’s token moved while the request was in flight, which still spares it
invalid_grant + Refresh token not found or invalid, or OpenAI’s refresh_token_reused	the value we sent is not the current one; the chain behind it is usually alive	count a strike, leave the account active
anything else (5xx, an unparseable body)	the refresh path may be broken	log at .error, change nothing

Three strikes condemn the account, and only if they are spread out: refresh_token! has nine call sites and several of them can present the same spent value within minutes of each other, so a second rejection within 15 minutes of the last is the same episode and counts once. Three strikes therefore take at least half an hour. A streak expires six hours after its most recent strike, and any new refresh token — from a successful refresh or a human re-authenticating on /inference — resets the count, because a new token is a new chain. The count lives on claude_accounts.stale_refresh_failures / last_stale_refresh_failure_at and is shown on the account’s Administrate record page.

A :stale rejection is also not retried. The 5-minute sweep’s retry ladder (2 + 4 + 8 minutes) exists for network blips; replaying a value the vendor has already rejected just spends three more requests to be told the same thing, and the ladder ends in an .error nobody can act on. The provider reports :stale as its own Result error kind and RefreshRuntimeAuthTokensJob logs it at .warn and waits for the next sweep, by which time another caller’s refresh may have moved the row on. That wait used to carry a liveness assumption it could not meet — it waited on a filesystem sync that a corrupt shared file made permanently impossible, the metronome that ran for three hours on 2026-08-22. There is no such sync now; what bounds the wait is the strike counter, which condemns a row nothing moves on and alerts when it does.

A genuinely dead credential still reaches a human, roughly half an hour later than it used to. A healthy account that lost a race no longer reaches one at all — which is the whole point, because that was 14 of the 15 accounts condemned over an eleven-day window in production (#530).

Never lose a token you just minted

The vendor spends the presented token the moment it answers, so between the 200 and the commit, the row holds the only copy of the credential chain. Persisting it is therefore the step that must not fail, and for a Claude row it is the only step: perform_claude_refresh! writes the pair in one update! and nothing follows it. There is no filesystem write that could raise after the vendor has already spent the old value, and no on-disk copy of the spent pair for anything to adopt back.

Codex still writes auth.json when the account is current, rescued and after the commit, and its sync answers the “do not move backwards” question from auth.json’s last_refresh, refusing tokens older than the ones it already holds.

The lock is re-entrant with the outer account.with_lock in RuntimeAuthProvider#recover_needs_reauth and in the sweep, so nesting is safe.

QuotaResetCheckerJob (every 15 min, Claude only) restores quota_exceeded accounts when ClaudeAccountQuotaSnapshot#windows_clear? says both windows have cleared: each window’s reset time has passed, or its utilization has dropped below 100% — except that a weekly window the API is still rejecting is never counted as clear, however far its counter has drifted. It then calls AuthOutageParkService.wake_parked_sessions! so the sessions that were blocked on those accounts resume in the same sweep — see When the pool runs dry.

The status column is sticky; the badge on Inference is not

ClaudeAccount#status is a durable column. Something writes quota_exceeded onto it and only the 15-minute sweep above ever writes active back. That makes it a claim about the past, and two things routinely leave it stale:

• A rotation can still outrun its own evidence. AccountRotationService#rotate_under_lock no longer stamps whatever it rotates past (A rotation is not evidence about quota), but a rotation that DID have quota evidence at the time still leaves a label behind, and Claude’s windows slide — the account is servable again well before the sweep next looks (see Auth recovery can rotate away from an account that was fine). What protects the next session meanwhile is that rotation validates a candidate at pick time.
• The sweep is not guaranteed to run. The deploy that froze every queue for ten hours (#426) froze every label with them.

So the page does not render the column unquestioned. ClaudeAccount#effective_status derives what an account presents from windows_clear? on its own latest snapshot — the same predicate the sweep restores on — and the account-level badge and the pool tallies both read that. It is the account-level counterpart of the staleness rule InferenceHelper#window_status_badge already applies per window: a recorded status describes the window that was open when the reading was taken.

The derivation runs one way, and only one. It softens quota_exceeded to active when the reading says the windows have cleared, and does nothing else: it never marks a healthy account, never touches needs_reauth — which only a human clears — and falls back to the column whenever there is no snapshot to judge by, which is every Codex account.

Every path that PICKS an account to spawn with keeps acting on the durable column — ClaudeAccount.available, AccountRotationService, and the wake sweep that is about to start a session — because a reading minutes old is not something to hand a session on. The column converges separately: InferenceController#auto_heal_accounts runs on page load as well as on refresh, from the same predicate, so looking at /inference is the other thing that can restore an account when the sweep is the thing that has stopped. Only the account — the sessions parked on it still wait for the sweep’s wake_parked_sessions! or their own timer, because resuming sessions is not something a page render should do.

Two decisions read the evidence instead, through ClaudeAccount.serviceable_for — the same windows_clear? rule, applied to the whole pool at once. See One predicate for “is the pool drained”.

A dead account tells you so

needs_reauth is the one account failure Zimmer cannot recover from. The refresh token is permanently invalid, the pool quietly stops drawing on the account, and everything keeps working — with a smaller pool. Nothing surfaces it: the failure is logged at .warn precisely so it does not page #alerts (a channel alert for a condition only a human can clear is noise), and recover_needs_reauth re-probes it forever without ever succeeding. The account just sits dead on /inference until somebody happens to open the page.

So Zimmer tells you — but it does not compose the message itself. When a ClaudeAccount crosses into needs_reauth, an after_update_commit callback emits the account_needs_reauth Zimmer event, and whatever ao_event Triggers watch that event fire. The one this deployment ships spawns a general-agent session holding the slack-workspace MCP server, with a prompt telling it to DM the operator, name the account, and say that fixing it means pressing “Authenticate” on /inference.

The indirection is the point, and it replaced a native DM that never arrived. That path was ClaudeAccount → AccountReauthAlertJob → AccountReauthNotifier → an operator DM → SlackService.send_dm, and it had three distinct ways to fail — an unset OPERATOR_SLACK_USER_ID, a bot without the im:write scope conversations.open needs, and a dedup key stuck from an earlier failure — each of which degraded to one .warn line and a false. Nothing surfaced any of them: the boot-time health check only ever looked at the Slack token and the channel id, never at the operator id. A deployment could report itself fully configured while dropping every DM it sent.

A Trigger cannot rot the same way. The notification is a session with a transcript you can read at /sessions, its prompt tells the agent to fall back to the alerts channel and say so if the DM will not send, and a fire that raises pages #alerts through AoEventTriggerJob’s existing failure handling. The trigger itself is a row at /triggers: editable, disable-able, and visible as a thing that exists.

The transition is latched during the save (after_update) and only acted on after the commit, which is not ceremony. reload nils @mutations_before_last_save, so a commit-time saved_change_to_status? answers “nothing changed” whenever anything reloaded the record between the write and the commit. RefreshRuntimeAuthTokensJob does exactly that: it wraps the refresh in an outer account.with_lock, and ClaudeAuthProvider#refresh! reloads on its failure branch to decide whether the error is :needs_reauth or :transient. with_lock opens a transaction without requires_new, so refresh_token!’s inner lock joins it and the commit callback does not run until that outer transaction commits — long after the reload. Asking at commit time meant the every-5-minutes sweep, the likeliest discoverer of a dead refresh token, never DM’d at all. A plain ivar survives reload; the dirty state does not.

A model callback rather than instrumentation at the sites that condemn an account, so no path can forget to alert — including the Administrate admin form, which no service-level hook would see. Two writes deliberately do not alert, and both fall out of that placement:

• Creation. after_update_commit does not fire on insert, so the credential-less account /inference seeds directly into needs_reauth stays silent. The human is on the page adding it.
• Recovery restores. recover_needs_reauth flips an already-dead account to active so refresh_token! is not status-blocked, then writes needs_reauth back with update_columns when the probe fails. update_columns skips callbacks, so that no-op round trip is silent — and the probe itself cannot condemn the account either, since recovery_probe: true returns before the permanent-failure branch. Without both, every recovery sweep would look like a fresh failure.

On top of that, the account itself carries the throttle: claude_accounts.reauth_alerted_at, checked and stamped by a single conditional UPDATE (ClaudeAccount#claim_reauth_alert_slot!) that admits one event per REAUTH_ALERT_THROTTLE (12 hours). Being one statement is what makes two workers condemning the same account in the same instant produce one event rather than two.

It lives in the database rather than in Rails.cache, where its predecessor lived, because a cache-backed suppressor fails open: with Redis unreachable, every crossing alerts. Under the old design that was a duplicate DM. Under this one it is a spawned agent session per crossing — and an account crosses into needs_reauth more often than it breaks, because plenty of machinery writes active back onto a dead row (the auto-heal sweep on /inference, a recovery probe that happens to succeed) and ensure_active_account! runs before every session spawn. The column is also readable after the fact, which a cache key never was.

Only a human re-authenticating releases it — ClaudeAccount#clear_reauth_alert!, called from ClaudeLoginDriver#capture! and its Codex twin, not from the status callback. That looks like the more obvious place and is a trap: the same machinery above writes active with a plain update! and no human involved, before every session spawn. Releasing there would drop the backstop moments before usable_candidate? re-condemns the same account — one spawned session per spawn attempt on a drained pool, which is the exact flood the window exists to prevent. The cost of the narrower rule is that an account the recovery sweep fixes automatically, which then dies again inside 12 hours, waits out the window before it can alert again.

A failed alert can never take down the auth path: the callback rescues anything the enqueue raises, and the firing job runs asynchronously on the triggers queue.

The circularity is real and is not solved, only bounded. Reporting that an account is unusable means spawning a session, and spawning a session needs a usable account. One dead account among six is fine. A pool where every account is dead cannot spawn the session that would say so — and the seeded trigger is priority rather than the spot that ao_event derives, precisely so the one session whose job is to report a dead pool is not also gated behind a healthy one. When the spawn fails anyway, AoEventTriggerJob#handle_fire_failure alerts #alerts, which needs no account at all. That is the floor: a channel post rather than a DM, but not silence.

An account can be capped without ever having been current

mark_quota_exceeded! used to fire only on the account that was current when a session hit a wall. An account that filled its weekly window while sitting idle in the pool was never marked: it stayed active, stayed in ClaudeAccount.available, and was the next thing rotation reached for — activate, fail on the first request, rotate again (#248).

Two changes close that, and they read the same predicate — ClaudeAccountQuotaSnapshot#seven_day_window_spent?:

• QuotaSnapshotService marks the account as the reading lands, whichever path took it — rotation, a /inference page view, the reset checker’s own probe. A reading that says the week is gone takes the account out of available there and then. It only ever moves an active account: needs_reauth is a different failure with a different recovery, and relabelling it would make an unusable pool look merely throttled.
• Rotation and bootstrap refuse a capped candidate at pick time, and mark it on the way past. That is what covers evidence recorded before the ingestion marking existed.

QuotaResetCheckerJob reads the same predicate to decide the account is not back yet, which is what keeps the healer and the marker from flipping an account between states on every sweep.

Bootstrap validates before it promotes

ensure_active_account! picks an account when nothing is current — a fresh install, a restarted worker, a pool that has just been re-authenticated. It used to take ClaudeAccount.available.first on faith, and ensure_fresh_tokens! swallows its own failure by design, so an account with a dead refresh token became current anyway and every session on the instance failed to authenticate until a human intervened (#239).

It now walks the pool in priority order and takes the first candidate that is not capped and whose token Anthropic actually honours. The probe is QuotaCheckService#check_with_token, not refresh_token!: it reads the rate-limit headers off a one-token request, so it can be run over candidates that may be skipped without spending a single-use refresh token. A candidate Anthropic refuses is refreshed once and probed again — a stale access token is the one refusal a refresh can fix — so the single-use token is spent only where it might help, never on a candidate whose credentials already work — and spent at most once per recorded refusal rather than once per spawn, so a pool whose only account has a live refresh endpoint and a dead subscription does not burn a single-use token every time a session starts. ClaudeLoginDriver#capture! applies the same probe and refuses the login on QuotaCheckService::Result#rejected?, because a login that produces a complete-looking token pair is another way an unusable account enters the pool. The probe’s model is the catalog’s Haiku, sent as its Messages API id; see Models.

The probe answers three ways, and only one of them condemns an account: Anthropic honoured the token, Anthropic answered and refused it, or the probe never got an answer (timeout, DNS failure, 5xx). The last is evidence about the network, not the credential, so it does not skip the account — reading an Anthropic outage as “every credential is dead” would park every session at once.

A successful probe is a live quota reading, so it is recorded as a snapshot rather than reduced to a boolean. That is what lets bootstrap refuse an account whose weekly window is spent but which nothing has ever probed — the case stored evidence cannot cover — and it leaves rotation with evidence about an account that has never been current.

Stored is not verified

A credential Zimmer is holding is in one of four states, and ClaudeAccount#credential_state is the one place that says which:

State	What it means	What the card says
:none	oauth_config is empty	”No credentials yet — authenticate to start serving sessions.”
:unverified	stored, and nothing has presented them to Anthropic since they were written	”Credentials stored, not yet checked against Anthropic.”
:verified	Anthropic answered a probe of this access token and served it	”Credentials verified against Anthropic 4 minutes ago.”
:rejected	Anthropic answered a probe of this access token and refused it	”Anthropic refused these credentials 4 minutes ago — re-authenticate to replace them.”, plus a red Token refused chip

The state is derived from two timestamps on the account row, credential_verified_at and credential_rejected_at, and the later one wins. They are written by ClaudeAccount#record_credential_probe! from probes Zimmer already takes — bootstrap’s candidate probe, rotation’s snapshot, the 15-minute ClaudeUsageSamplerJob sweep, the reset checker, a /inference refresh, the switch validation, and the login capture. Nothing here adds a network call, nothing polls on render, and nothing spends a single-use refresh token (#242).

Two rules keep the verdict honest:

• An unreachable Anthropic records nothing. Same asymmetry as the promotion probe above: a timeout, a DNS failure or a 5xx is evidence about the network, not the credential.
• The verdict is about the token, not the account. Writing a different access token — a refresh, a human re-auth — clears both timestamps and the row falls back to :unverified. So a refusal cannot outlive the credential it was about: the account whose refresh repairs it is back in the pool on the next save, with nobody probing anything.

has_valid_config? is unchanged, and deliberately so. It answers “is there anything stored”, which is what the Codex provider’s filesystem adoption, rotation’s current-account branch and the two rake listings are actually asking. What changed is that the UI stopped reading it as “these credentials work” — the sentence “Credentials stored. Re-authenticate to replace them.” sat over an account answering 401 for the whole of the 2026-07-31 outage, and is what made it so hard to diagnose from the page (#239).

A recorded refusal is also a reason for ensure_active_account! to stop short-circuiting: the current account no longer keeps its place on has_valid_config? alone, and a row Anthropic has refused falls through to the bootstrap path, which re-probes candidates and promotes one that works. The recorded verdict decides nothing on its own — the fresh probe does — so an account that started working again is promoted straight back.

When nothing can serve, the page says so first

When no account in a runtime’s pool can serve a session, /inference opens with a red banner naming the pool, accounting for every account in it (“1 refused by Anthropic, 2 out of quota”) and saying what clears it: a quota wall lifts itself, anything else is waiting on the human reading the page. It is re-rendered by the refresh actions along with the cards, since a refresh is exactly when the answer changes.

The predicate is ClaudeAccount.any_serviceable_for? — the same one the park decision, the recovery coordinator and /health’s auth card ask, not a second opinion. See One predicate for “is the pool drained”. Before this, the page reported “1 Active” while zero accounts could serve a session.

What /inference reports for the pool

Anthropic meters two windows per account, 5-hour and 7-day, and the page shows both — per account, and averaged across the pool.

The pool’s 5-hour average is labelled “Avg 5-Hour Utilization (servable accounts)” because it is not a plain average of the 5-hour counters. An account whose 7-day window is spent — status rejected, or the counter at its cap — cannot serve a request however much 5-hour headroom it reports, so it is left out of that average, and its card says “Left out of the pool’s 5-hour figure” under the 5-hour bar. An account at 29% on 5 hours and 100% on the week would otherwise pull the headline down and advertise pool headroom that nothing can spend.

It is left out rather than counted at 100%, and the difference is not cosmetic. The spot gate reads this average as 5-hour capacity consumed and paces against it, so substituting 100% put a floor on the pacing curve that measured nothing the fleet did and that no amount of idling could lower — see Read across the whole pool and #693. The account still says it cannot serve, on the 7-day figure, where it reads 100% honestly. When every account’s week is spent there is no servable account to average and the figure falls back to the whole pool, which then reads 100% — reaching the right answer without substituting onto anything.

The correction runs one way. The 7-day average takes the 7-day counters as they are, because the weekly window subsumes the 5-hour one: an account at its 5-hour cap is idle for minutes and then serves again. A window whose reset time has passed reads as 0% on both axes, since the sliding window has cleared and the last number Zimmer recorded for it is stale.

A status expires with its window for the same reason a counter does. Past its reset time a card shows the green “Window reset” line and no status badge at all, rather than the rejected the reading recorded — that status described a window that no longer exists, and leaving it up puts a red badge beside the 0% the same snapshot renders. Before the reset the card names the wait instead: “Resets in 1d 4h”, and “Resets in < 1m” through the last minute, which has no whole unit left to report.

That wait is a value as of the render, not a live countdown. /inference is a static page with no poller on the cards, so every figure — the wait, the bar, the badge — is the reading as it stood when you loaded it and stays where you left it. “Resets in < 1m” is therefore the one that goes stale fastest, on a tab left open. Reload for the same snapshot re-read against the clock, or hit Refresh for a live probe.

There is one definition of “spent”, and it lives on the snapshot: ClaudeAccountQuotaSnapshot#seven_day_window_spent? — the API status is blocking (allowed and allowed_warning serve; anything else, including a value Anthropic adds later, blocks), or the counter has reached the cap, in both cases only until that window’s reset time passes. The page reads it to decide what an account contributes, QuotaSnapshotService reads it to mark an account quota_exceeded as the reading lands, rotation reads it to refuse a candidate, and QuotaResetCheckerJob reads it to decide the account is not back yet. They used to disagree at the rejected-but-under-100% edge, which meant the page called an account spent while the healer called it clear.

Its counterpart is #windows_clear?, on the same model for the same reason: QuotaResetCheckerJob restores an account on it, InferenceController#auto_heal_accounts heals one on it, and ClaudeAccount#effective_status decides what badge /inference renders from it. It applies the status-outranks-the-counter rule to both windows. The 5-hour one was counter-only until the badge started deriving from this predicate, at which point a window the API reports as rejected at 90% would have rendered “Rejected” beside an account badge reading “Active”.

pool_utilization_5h itself is display-only — no scheduler or rotation path reads it. The underlying snapshot numbers are not: QuotaResetCheckerJob and InferenceController#auto_heal_accounts flip accounts back to active from them, and QuotaSnapshotService flips them out.

Rotation is triggered by matching an English error string

ApiErrorRetryService::ACCOUNT_QUOTA_LIMIT_PATTERN:

/hit your\b.*\blimit\b.*\bresets\b/i

That regex, run against the transcript message text, is the sole trigger for the entire multi-account rotation feature. That trigger is Anthropic’s prose: a regex over transcript text, with no HTTP status or structured error code behind it.

On 2026-06-14 the CLI changed “hit your limit” to “hit your session limit,” which the regex happens to still match. A previous wording change did not, and rotation silently stopped firing. The session fell through to the transient-rate-limit path, retried six times against an already-capped account, exhausted its retries, and failed, with no log line saying rotation should have fired. The failure mode is silent. Tracked in #53.

Mid-run auth loss

AuthRecoveryService watches the transcript for an authentication failure and, when the turn ends on one, hands the decision to AuthRecoveryCoordinator rather than treating the exit as a normal pause. Only the last main-chain user or assistant entry is eligible: runtime bookkeeping entries are skipped, as are sidechain entries. An older auth error followed by successful assistant output is a completed turn and is not recovered. This is the same terminal-conversation rule used by the API-error backstop, and it matters for status-summary forks that import an existing transcript before appending their own successful answer. Recovery is bounded by MAX_RECOVERY_ATTEMPTS attempts within CONSECUTIVE_WINDOW (15 minutes).

It recognizes the failure two ways, and the order matters:

1. The error type. AUTH_ERROR_TYPES — authentication_failed, oauth_error, oauth_org_not_allowed — matched against the transcript entry’s error field. This is the machine-readable half of the signature and the half that does not move when the prose does.
2. The prose. AUTH_RECOVERABLE_ERROR_PATTERN — not logged in, please run /login, failed to authenticate, oauth/refresh/access token … expired|invalid|revoked, invalid_grant, oauth_org_not_allowed, disabled Claude subscription access — for the entries the runtime records with an empty error type, which is how "Not logged in · Please run /login" is recorded.

An entry the API typed as retryable (api_error, rate_limit_error) is never claimed on prose: a transient upstream failure that happens to say “Authentication failed: 401 from gateway” belongs on the backoff path, not on a path that spends an account rotation. The structured type wins both ways.

The prose net is deliberately wide, because here a false positive costs one rotation and a false negative costs a lost turn. That trade does not travel: SessionStatusSummaryHarvestJob asks a different question — is this “summary” a refusal? — where a false positive discards a real summary, so it spells out its own two narrow patterns instead of importing this constant.

Why the type is read first

On 2026-08-20, Claude Code 2.1.237 ended a turn in production session 6412 with

{"isApiErrorMessage":true,"error":"authentication_failed",
 "message":{"content":[{"type":"text",
   "text":"Failed to authenticate: OAuth session expired and could not be refreshed"}]}}

and exited 1 — its “turn finished, awaiting input” convention. The prose matched neither half of the pattern Zimmer had, no classifier claimed the entry, and handle_exit logged “Process exited successfully” and parked the session as needs_input. A human’s message sat unanswered, and the only trace was in the transcript. The account pool had already done its job: the refresh token for the identity that session was holding had been rejected with invalid_grant seven minutes earlier and the account marked needs_reauth. What failed was recognizing the runtime’s report of it.

An organization refusal benches the account

Most auth failures are about the credential: the token expired, was revoked, or the child is holding an old copy. oauth_org_not_allowed is about the account. Claude Code accepted the token and refused the organization behind it:

Your organization has disabled Claude subscription access for Claude Code · Use an Anthropic API key instead, or ask your admin to enable access

A refresh does not fix that, re-seeding does not, and waiting does not. The token still passes the Messages API probe, so the ordinary re-seed branch would hand the same account straight back. And an auth rotation leaves the outgoing account active, so the next session would get it and fail the same way.

So AuthRecoveryService.access_disabled? picks this subset out. It reads the oauth_org_not_allowed type, or the prose on an untyped entry. Before anything else is decided, the coordinator benches the identity the session was spawned with: ClaudeAccount#disable_access! writes needs_reauth and stamps access_disabled_at. Then it skips the probe and the re-seed and rotates, or adopts if the pool already moved. The session resumes on the healthy account. If nothing healthy is left, the benched account also stops being current, because the spawn path hands out the current account’s token without reading its status, and this token still authenticates.

needs_reauth takes the account out of .available and .serviceable_for, and alerts a human through the account_needs_reauth event. access_disabled_at keeps it out. The needs_reauth sweep restores any account whose refresh still works, and an org-disabled account’s refresh still works, so needs_reauth_recovery_candidates and recover_needs_reauth both skip a benched account. The bench is lifted by a save that writes active through callbacks: a human login on /inference, or an operator changing the status in /supervisor. The sweep’s update_columns restore cannot lift it.

Benching and rotating off the account logs at WARN. The park logs at ERROR only when nothing healthy is left to rotate onto.

A turn that dies on an API error can never look finished

Reading the error type makes this wording classifiable. It does nothing about the next one, so handle_exit also asks a question that has no prose in it at all.

ApiErrorRetryService#terminal_api_error answers did this turn die on an API error? — is the last conversational entry in the transcript an isApiErrorMessage. Only user and assistant entries count on either side of the question: the runtime writes last-prompt, atis-latch, attachment and queue-operation bookkeeping after the final message, and isSidechain entries belong to a subagent whose failure does not end the main turn. Unlike every other transcript reader it ignores api_error_last_checked_line — that cursor exists to stop a handled error being retried twice, and a terminal error is by definition the one that just ended this turn.

It does not filter by classifier, and that matters. handle_exit asks it last, after every specific branch has looked at the same exit and declined, so an answer means nobody is handling a turn that plainly died — which covers a wording nothing recognises and the case where a classifier does recognise it but has already spent its cursor on it. That second case is reachable: a 5xx is retried, api_error_last_checked_line advances past it, the respawn writes nothing and exits 1, and every branch now says “not mine” about a transcript whose last word is still that 5xx.

When it answers yes, the session fails, loudly, and the prose goes into the session log under the terminal_api_error failure reason. Whether it also alerts is the one thing the recognised/ unrecognised distinction decides: an unknown wording goes to UnclassifiedFailureReporter (deduped per runtime, so a fleet-wide wave is one Slack message), while a recognised one has already been through its own classifier and is a dead turn rather than an unknown failure mode. Failing is the honest verdict either way — the turn is over, its work did not happen, and the prompt is sitting unanswered — where needs_input said the opposite of all three.

The backstop fires once per dead turn: it records the transcript line it fired on in metadata["terminal_api_error_line"], so a resume that writes nothing new leaves the same entry terminal without re-failing the session and re-alerting on it.

So the next time Anthropic rewords an error, the cost is an alert naming the new wording, not a lost message someone finds by reading a transcript.

A safeguards rejection fails on purpose, and does not page

One wording the backstop knows is not a stale classifier but a deliberate stop. When Anthropic’s safeguards refuse the request a turn makes, Claude Code records a synthetic assistant entry with error: "invalid_request" and exits 1 — the same shape as every other dead turn above:

API Error: Opus 5's safeguards flagged this message (https://www.anthropic.com/legal/aup). This
sometimes happens with safe, normal conversations. Claude Code can't respond to this message with
Opus 5.

Try rephrasing the request in a new session or change your model.

Learn more: https://support.claude.com/en/articles/16049681

Details: `[reasoning_extraction]`

Request ID: req_…

ApiErrorRetryService::SAFEGUARDS_FLAGGED_PATTERNS matches the verb (safeguards flagged this message) and the policy link (anthropic.com/legal/aup), and nothing else: the model name changes with the session’s model, and the bracketed Details: token names whichever classifier fired.

It is recognised, so it never pages as an unknown wording. It is not retried, and that is the decision rather than an omission. The refusal is a 400 on the request the turn just made; a plain resume sends the same conversation back on the same model, which is the request that was refused, and the backoff ladder would spend the session’s shared API-error budget learning that six times. The message’s own remedies — rephrase in a new session, change the model — each change what the session is, and the model is the human’s choice. So ProcessLifecycleManager#handle_safeguards_rejection fails the session at once under failure_reason: safeguards_flagged, with the CLI’s text (guidance included) in exit_status and the session log, and Session#failure_summary says the same thing in one line on the homepage and in the push notification. The session stays resumable: change its model with the button on the session page and resume, or start a new session with the request rephrased.

The first one, on 2026-09-20, refused Zimmer’s own PR-merged notification to production session 19243 and paged #alerts as an unclassified terminal API error (#1217). The page was right — the wording was unknown — and this is the classifier it asked for.

The recovery decision tree

Everything below runs under a pool-wide advisory lock (ClaudeAccount.with_pool_lock, namespace 0x415F4143, keyed per runtime), so N sessions hitting the wall at the same moment take these branches one at a time instead of each starting its own rotation.

The branch is chosen by comparing the pool’s current account against metadata["auth_identity_email"] — the identity the session’s process was spawned with, recorded by AgentSessionJob before each spawn and re-recorded whenever the coordinator or the quota path moves this session onto a new account. It is a per-session record, so it can lag: nothing writes it when another session rotates the pool, or when an operator switches accounts from the Inference page. The consequence is bounded and named under a stale spawn identity.

The spawn environment also records a SHA-256 fingerprint of the access-token generation actually handed to the child. It contains no token value; it only answers whether the DB generation moved since this process started — which is what tells “the token I am holding is one refresh behind” from “the token I am holding is the one that failed”.

An adoption costs nothing against the retry budget: it is another session’s rotation doing this one a favour, not an attempt this session made, and charging for it would park a healthy long-running session for the fleet’s activity. Adoptions are separately capped at MAX_FREE_ADOPTIONS (3) per window, after which they start costing budget — a free retry that never converges is the same unbounded loop the attempt cap exists to stop.

Why a Claude session’s access token is probed before rotating

“Not logged in” is the runtime’s word for both your token is dead and you are out of quota, and those two want opposite instructions in the outage banner. Per-session credentials add a third case: the process can hold the access token from before some other process refreshed the same DB account. The account is healthy, but that already-running process cannot see its replacement env value.

Recovery therefore starts with the same one-token Messages API call that supplies quota readings. It does not spend the single-use refresh token. A clear reading proves both that the DB-held access token authenticates and that the account can serve; when its fingerprint differs from the failed process’s, Zimmer keeps the account and re-spawns the session with the newer token (:reseeded). A legacy process with no fingerprint gets one such re-seed, and the replacement records its generation at spawn. If that exact generation reports auth failure again, recovery rotates rather than spending all three attempts on the same value. If Anthropic refuses the stored access token, Zimmer refreshes once and probes the replacement; a repaired, clear account is likewise re-seeded. Rotation is only reached on live quota evidence, failed repair, or a repeated failure of the same token generation.

This ordering is load-bearing. A refresh replaces the account’s access token, invalidating the value already present in every running session environment. Using refresh as the first “probe” caused an auth-recovery cascade: one stale process refreshed the healthy current account, every sibling began reporting “Not logged in”, and each sibling refreshed it again before parking because the other accounts were quota-capped. /inference correctly showed the current account with room throughout; recovery made that room unreachable to the processes it had just invalidated.

Codex and Pi retain refresh-before-rotation classification: the Codex CLI can own a newer refresh chain on disk, and a permanent OAuth failure marks the outgoing account needs_reauth rather than letting rotation relabel it. For a Claude row nothing else can hold the chain, so recovery does not spend a refresh to classify the outgoing account; the five-minute sweep condemns a dead refresh token and alerts when it does. For every runtime, whether an account is labelled quota_exceeded is decided by its own reading and the rotation’s reason, not by the fact a rotation happened. The pool’s resulting shape is what AuthRecoveryCoordinator#park_reason_for_pool reads — through serviceable_for, so it reads the readings rather than the labels. The distinction is made on evidence, not on prose.

This coordination adds no new string matching; the fragile pattern in this subsystem is still the /not logged in|please run\s*\/login/i match above.

Why the attempt bound is time-based

The bound is time-based, not success-based, and that distinction is the whole reason the service terminates. A re-spawned Claude Code process spends its first 10–15 seconds connecting MCP servers before it makes the API call that reports “Not logged in”, so it clears any short liveness check even when the credential it was handed is dead. Treating that liveness as recovery success — and resetting the attempt counter on it — made the counter oscillate 0 → 1 → 0, so MAX_RECOVERY_ATTEMPTS could never be reached. Production session 684 logged retrying 1/3 115 times over 35 minutes, re-spawning the CLI into the same auth wall roughly every 18 seconds, and surfaced nothing to the user but a wall of Not logged in · Please run /login. Attempts now accumulate and are only forgiven by elapsed time, so an unrecoverable identity costs three re-spawns instead of an unbounded loop.

This used to be three walls, not one

Before the coordinator, recovery meant “re-inject the current active account” unconditionally. When that account was itself the problem, each attempt wrote the identical dead credentials back to disk and re-spawned into the identical wall — so the user saw Not logged in · Please run /login three times — and the session then parked with AUTH_UNRECOVERABLE, telling them to re-authenticate when the actual remedy was to wait for a quota window.

Production session 657 is the trail. At 11:46:39Z it logged “Not logged in detected on successful exit - attempting auth recovery” → “Auth recovery limit reached (3 attempts) — failing” → parked AUTH_UNRECOVERABLE for an hour; by 11:50:12Z it was back at “retrying 1/3”, refreshing to the same account. Across the whole incident it logged not one Account quota hit — rotated to … line: the rotation path never ran, because the auth signature is checked before the API-error path (most-recent-error-wins) and shadowed the weekly-limit message sitting in the same transcript.

When the pool runs dry

Three exits mean Zimmer has no runway left: a quota hit with no rotation target (AccountRotationService#rotate! → { success: false, reason: "no_available_accounts" }), an auth recovery whose rotation finds the same thing, and an auth recovery that exhausts its retry budget. All route to AuthOutageParkService, which:

1. Records auth_outage_reason / auth_outage_parked_at on the session — which renders the amber outage banner on the session page — and puts it to sleep, in the same write. A session still running is marked pending_sleep and carried needs_input → waiting by the pause callback; one already at rest is slept outright, inside one transaction with the record. Either way it lands in waiting rather than needs_input, so the heartbeat sweep anchors its cadence instead of nudging it back into the same wall.
2. Writes a session log naming the outage.
3. Sends a push notification.

The dormancy and the record of it are one statement because a park that marked a session to sleep and then failed to say why produced a session dormant in waiting with no exit_status and no park reason — #608. The announcement in (2) and (3) runs after the park is committed and is separately rescued, so a failure there cannot report a parked session as unparked.

One predicate for “is the pool drained”

Parking a session is a claim about the whole pool, and it is expensive: the session sleeps, and a human reads “Quota exceeded across all Claude Code accounts”. So the claim is made on the accounts’ own readings, not on their labels.

ClaudeAccount.serviceable_for(runtime) is that question, asked once. It takes every account that is active or quota_exceeded and holds credentials, and keeps the ones whose effective_status is active — so an account the column calls exceeded while its own reading says both windows are clear counts. An account with no reading is taken at its label, which is every Codex account, so for a pool with no snapshots this reduces exactly to .available.

One account is dropped whatever its label and whatever its windows say: one whose stored access token Anthropic has answered and refused (credential_rejected?, see Stored is not verified). That row is the 2026-07-31 state — active, credentials stored, counted by every “is the pool dry” surface, and a 401 on every request made with it. Only an answered refusal drops it, and the verdict retires the moment a different token is written, so neither an Anthropic blip nor a repaired account can strand the pool here.

Only a reading the label has not already answered. The reading has to be newer than the account row’s last write, or the column stands. A label written after the newest reading was written by something that knew more than the reading does — a runtime-observed quota refusal whose follow-up probe failed, say — and overruling it would resurrect an account that every spawn path still refuses and that the healer’s own fresh probe will decline to restore. Where they disagree in that direction the predicate degrades to exactly .available, which is the safe floor.

Three callers ask it, and they are the three that must not disagree:

Caller	What it decides
AuthRecoveryCoordinator#park_reason_for_pool	whether an outage is QUOTA_EXHAUSTED (“wait for the reset”) or AUTH_UNRECOVERABLE (“a human must re-authenticate”)
AuthOutageParkService.pool_confirmed_empty?	whether an undelivered turn may be parked at all
HealthMonitorService#auth_health	the serviceable_accounts figure on the health report and the /health card
InferenceHelper#dry_pool	whether /inference opens with the dry-pool banner

That last row is why this exists. On 2026-08-23 the parking decision concluded at 02:06Z that the pool was empty and put four sessions to sleep, while auth_health reported “3 Claude accounts available” at 02:13Z — two code paths in one app contradicting each other about one fact, because both were reading a sticky column minutes apart and the healer moved it in between.

auth_health reports both numbers rather than swapping one for the other, because they answer different questions and the gap between them is itself the diagnostic. available_accounts is the column: what a session can be spawned on this minute, since every path that picks an account reads it. serviceable_accounts is the predicate above: what the park decision sees. Reporting only the column is the contradiction described here; reporting only the evidence would be its mirror image, a healthy card over a pool nothing can spawn against. A third figure, refused_credential_accounts, says how many of them hold a credential Anthropic has refused — the same distinction /inference’s banner draws, carried to the card and, through the same report, to get_system_health. Together, 0 available / 3 serviceable says precisely what is happening — the pool is recovering and the reset checker has not caught up — and the card degrades to warning in that state rather than claiming health.

The banner’s recovery estimate is derived the same way. AuthOutageParkService#earliest_pool_reset drops any account whose reading says its windows are clear, and lets the accounts that ARE blocked set the estimate. A clear window is not waiting for anything, so its reset stamp is when the counter next rolls over rather than a recovery time — reading one anyway is how the 02:06Z park told four sessions their pool came back at noon. Contributing “now” instead would be the opposite error: it would win the pool-wide minimum and promise a recovery the blocked accounts cannot deliver.

Waking is deliberately the opposite. AuthOutageParkService.runtime_has_available_account? and QuotaAvailabilityMonitor both read the durable column, because resuming a session against an account the column still calls exceeded starts it into a pool that will not serve it — QuotaResetCheckerJob restores the column and then calls both in the same tick, which is what keeps them in step. Parking must not sleep a session over a stale label; waking must not start one over a fresh reading the rest of the pool has not adopted yet.

Nothing is scheduled. A park used to create a one-time wake-up trigger per session — the same reuse_session + last_session_id shape wake_me_up_later uses — carrying a retry time derived from the quota snapshots, plus a backoff ladder and a jitter window to keep a whole parked cohort from waking as a herd. That machinery scaled with the number of PARKED SESSIONS rather than with the number of outages: production carried dozens of Auth outage retry for session #N at … rows at a time, the large majority of the whole trigger list. And a wall clock knows nothing about which parked session matters most, so whichever timer fired first won, regardless of precedence.

Two mechanisms wake a parked session now, and AuthOutageWakeAuthority is the one place that says which parks belong to which. Everything that starts a park, and every surface that tells a human or an agent who is coming for one, asks it.

Population	Woken by	Why
spot — AuthOutageWakeAuthority::FLEET	the quota_available trigger event → one fleet-maintenance session running the awaken-waiting-sessions skill	spot work is exactly the work whose ORDER matters when quota is scarce, and the skill is what reads precedence, the spot thresholds and the concurrency ceiling to decide who starts. Both artifacts ship in Zimmer’s own catalog, so the seeded trigger resolves on a standalone install; a test asserts the root it names exists
priority — AuthOutageWakeAuthority::SWEEP	AuthOutageParkService.wake_parked_sessions!, from QuotaResetCheckerJob every 15 minutes	priority work is never gated on quota, so making it wait for a spawned session to take its first turn would be a regression — and there is no ordering question to get wrong

The split is old; the single answer is not. It used to be a paragraph, restated in four comments and re-derived from the scheduling class at each of them, while the fleet-side policy — which lives in tadasant/tadasant-internal, not here — believed it owned the whole parked population. Both sides claimed the priority parks, and the skill’s own collision check answered by refusing to wake anything, handing the recovery to a sweep that ranks nothing (#617). Two things follow from that, and they are the shape of the fix:

• The owner is derived, never stored. A park does not stamp its owner into metadata. A session’s scheduling class can be changed while it is asleep, and a stamped owner would then name a mechanism that is no longer looking for it — a park owned by nobody, which is #655’s shape. Derived, a reclassified session simply changes hands on the next sweep.
• The boundary is stated where the other side reads, not only in Zimmer’s comments. A parked session’s get_session output carries a Woken by: line, and its quick_search_sessions row says the same thing without costing a detail read. For the half the fleet wake must not touch, those lines say so outright.

Neither wakes a session that is also asleep on a wake-up somebody chose. A park and a pause are different states that happen to share waiting: the pool recovering answers the park and says nothing about the pause. The sweep skips those sessions before its spot/priority branch, so a paused spot park is not counted toward the fleet wake it must not be started by, and a paused priority park is not resumed — which matters twice over, because that resume goes through resume! and would have consumed the pause without a trace. See A pause outranks precedence.

QuotaAvailabilityMonitor owns the event. It runs in the same QuotaResetCheckerJob pass, right after the accounts are restored, and asks one question: can the pool serve a request at all — is there an account that is neither quota_exceeded nor waiting on a human to re-authenticate. It reads the durable column (accounts.available), which is the same thing the wake sweep it feeds reads, and deliberately not the evidence-based predicate a PARK stops on — see One predicate for “is the pool drained”.

That rising edge is necessary but not sufficient. Before firing, the monitor asks SpotGateService whether a quota window is holding spot work, because starting spot work is the entire job of the session this event spawns. An at_utilization_limit hold means there is nothing to hand out, so the event is deferred: the stored level stays false and the next sweep asks both questions again. Nothing is spent and nothing is lost — the hold lifts on a window rolling over or on the fleet’s burn falling, neither of which needs this event to happen first. Parked priority sessions are unaffected either way, because the same sweep resumes them directly and the gate never holds them.

fleet_at_cap is deliberately not a deferral reason, though it zeroes the woken session’s headroom just as effectively. A window’s hold moves on the window’s clock, slower than this fifteen-minute sweep, so observing it once is good evidence it will still be there in a minute. Cap contention moves on a session’s clock, much faster — a slot frees whenever anything finishes — so a fleet habitually at its cap would show fleet_at_cap to every sweep while ordinary held spot sessions took the freed slots on their own ten-minute ladder, and the outage-parked sessions, whose only wake path this is, would starve behind them. It is also the honest scope: this event is the quota pool recovering, and cap contention is not a quota condition.

The two readings drift apart because they measure different things, which is the whole defect (#611). An account goes back to available when Anthropic’s own window clears; the gate compares the pool’s spend against the operator’s reserve and pacing curve. A pool whose accounts are all unflagged while its weekly spot budget is spent reads available and held at the same instant — which fired 27 fleet sessions in ten hours on 2026-08-22, every one of them reading HELD / at_utilization_limit, waking nobody, and spending a priority slot against the very window whose utilization was holding the gate.

The check fails open in both layers: SpotGateService already allows a session on any condition it cannot evaluate, and a raise on the way to asking is treated the same way. A spurious fire costs one session, which then re-reads the gate for itself; a suppressed one costs every parked session its only wake path.

A fire that delivers no session — nothing listening, every fire raised, every one burst-suppressed — puts the edge back (QuotaAvailabilityMonitor.rearm!), so the next sweep fires again rather than spending the one recovery the parked sessions were waiting for.

The event is an edge, not a level. AppSetting#quota_pool_available stores the last observed level, and the event fires only on false → true; a level would be true on every sweep for as long as the pool stayed healthy and would spawn a fleet session every fifteen minutes forever. NULL means nobody has looked yet, so the first observation records the level and fires nothing — a deploy landing while the pool happens to be healthy is not a recovery. A pool that cannot be read leaves the stored level alone, because recording an unreadable pool as an outage would make the next successful read fire a recovery nothing recovered from.

Sampling alone is not enough to see the falling edge. check! runs every fifteen minutes, so an outage that opens and closes inside one tick is never observed as unavailable — the recovery is then not an edge, nothing fires, and everything parked in that window waits forever. So the park itself records it: park! calls QuotaAvailabilityMonitor.record_unavailable!, which is both the earliest moment Zimmer has positive evidence the pool is empty and the most certain.

The sweep can also ask for the wake outright (request_wake!), and this is the second half of its job: it starts the parks it owns, and it speaks for the ones it does not. Every pass counts the fleet-owned parks it found eligible and left alone, and hands that number to request_wake!. Two different populations end up in the count, and this is the only path either has:

• An auth park, for which no edge fires at all — it is woken by the pool’s credentials changing, which happens with accounts.available true throughout, so check! never sees a false→true.
• A park the fleet session did not reach on the edge that did fire, because the ceiling or the windows ran out first. The edge is spent, and nothing re-examines it until the pool exhausts and recovers all over again. #655 is a spot session found 33 hours past its own recorded recovery time in exactly that position.

It is a request and never a wake: the fleet session re-reads the gate, the ceiling and the ordering for itself, so the sweep never decides which spot session starts. Only Claude Code parks are counted into it, mirroring the scope record_unavailable! keeps: quota_available is announced against one global level that only ever reads the Claude pool, so a Codex park asking for it would spend the Claude edge on a recovery of a pool it was never blocked on — and the fleet wake scopes its own enumeration to the recovered runtime, so it would not have started that session anyway.

An unspent level is claimed exactly as check! claims one — and, because this path is handed a count rather than a reading, it first refuses outright on a pool that is confirmed able to serve nothing (an unreadable pool falls through, per the fail-open rule above). A spent level may be announced again once it has stood for ANNOUNCEMENT_STALE_AFTER (1 hour) — the same conditional UPDATE claim against a different predicate, which renews the stamp so the next re-ask is another hour away. What must not happen is a re-arm: putting the level back to false would make the next pass of the very same cron read it as a fresh rising edge, and one parked session would spawn a fleet session every fifteen minutes for as long as it stayed parked, each burning the quota that just recovered.

A re-announcement also defers on any held gate, not just the window’s. The asymmetry is deliberate: fleet_at_cap must not suppress a recovery’s one edge, for the reason two paragraphs up — but a re-announcement is not that edge, it is a fifteen-minute re-ask standing behind it, so deferring it while every slot is full costs an hour at most and saves a fleet session that could have started nothing. The level and the job that spends it are written in one transaction for the same reason: a job that ran before the level committed would find nothing delivered, re-arm against a stale false, and silently lose the edge.

auth_outage_pool_recovers_at survives as an estimate for the banner, and only for a quota park: QuotaResetCheckerJob clears an account only when both windows are clear, so an account frees up at the later of its two future resets and the pool at the earliest such account. Nothing reads it back and nothing fires at it. An auth outage has no published reset clock, so it records nothing.

The park has to survive the paths that do not know about it

Everything above assumes the code that stops a session knows the pool is what stopped it. Three paths did not, and on 2026-08-20 a pr-merge-gate session (#6597) escaped through all three in eight seconds — parked correctly at 04:19, and back in needs_input by 09:00:42.

1. The resume left a window for the orphan sweep. resume_parked! transitioned the session to running and then enqueued its job, so for a moment it was running with a blank running_job_id — which CleanupOrphanedSessionsJob calls “DEFINITELY orphaned” with no grace period. The sweep landed in that window, reaped the resume, and replaced it with a resume-monitoring job pointed at a stale pid. The resume now stamps pending_follow_up_prompt inside the same transaction as the transition (the marker that sweep already honours) and records running_job_id as soon as the job exists — the same ordering Session#deliver_follow_up! uses, and for the same reason.

2. The reconnect could not tell our process from a stranger. ProcessLifecycleManager#resume_monitoring adopted a pid on Process.kill(0, pid), which answers “some process holds this number”, not “the process we spawned is still there”. It confirmed a recovery onto a pid that had been SIGKILLed nine seconds earlier. AgentProcessLiveness already recorded the boot id, PID namespace and start-time ticks needed to tell those apart, but was wired only into the spawn guard; .adoptable? is the read-only half, and refuses :dead and :recycled while standing down on :unknown so macOS development still reconnects.

3. The exit paths read a dead process as a finished turn. With the adopted process gone, the monitoring loop’s fallbacks answered with pause! — needs_input — while active_follow_up_prompt still held the recovery turn Zimmer never delivered and the pool was still empty (two status-summary forks parked quota_exhausted three and eleven minutes later). AuthOutageParkService.park_undelivered_turn! now guards those exits: an undelivered turn plus a pool with no available account is the outage, so it parks into waiting instead.

The tempting test for “undelivered” is that active_follow_up_prompt is still set, and it is the wrong one. AgentSessionJob removes that key in exactly one place — the :needs_input branch of the exit decision — and the paths this guard protects are the fallbacks, which never clear it. So a turn that ran to completion and exited through one of them still carries the marker; on the end_turn-plus-dead-process fallback that marker sits next to the strongest evidence available that the turn did complete. Parking on it would sleep a finished session and then nudge an agent with nothing left to do.

park_undelivered_turn! therefore asks for positive evidence, and refuses on every one of these:

It declines when…	Because
the prompt appears in the persisted transcript	the turn ran — the same comparison TranscriptPollerService makes to decide a follow-up landed. 6597’s transcript never grew past the 115 messages it held before the park
the session is not running	a user pause terminates the process before transitioning, so these exits are reachable for a session a human already stopped
paused_by is "user"	same reason, stated directly — putting the session back to sleep would undo their decision
auth_outage_reason is already set	two of the three call sites can be reached in one pass through the monitoring loop, and a double park sends two push notifications for one stop
the pool could not be read	an unreadable pool is not an empty one. .runtime_has_available_account? rescues to false meaning “do not wake”, which is conservative; the same false here would mean “park”, which is not — and a runtime with no accounts at all would then park, time out, finish and park again for as long as it lived

The sweep wakes a batch, not the fleet

wake_parked_sessions! resumes on “the runtime has an available account again”, which is a fact about the pool: it becomes true for every session parked on that pool in the same instant, and the sweep resumed every one of them in a single pass. One restored account therefore put the whole parked population back onto a pool with one account in it, which they re-drained in seconds and re-parked together.

So a sweep resumes at most MAX_WAKES_PER_SWEEP (5) per runtime, oldest park first, and logs how many it held. Per runtime because the hazard is per pool: a fleet of claude_code parks must not hold back the codex sessions whose own pool just recovered. The throttle closes its own loop — the next sweep is 15 minutes away and re-reads the pool, so if the batch that went first drained it again, nobody else is woken. Held sessions lose nothing: each leads the next sweep’s queue. The ordering is a lexicographic sort over auth_outage_parked_at, which is why that stamp is written explicitly in UTC.

Spot sessions are skipped entirely — see the table above. They stay parked with their outage metadata intact, which is exactly what the fleet wake needs to find them.

Which of the two reasons a park gets is decided by the pool’s shape, not by which code path arrived there. AuthRecoveryCoordinator#park_reason_for_pool answers QUOTA_EXHAUSTED only when nothing is serviceable and at least one account is quota_exceeded (waiting genuinely helps), and AUTH_UNRECOVERABLE otherwise — including when the pool can still serve and the runtime is rejecting it anyway, which is a credentials problem a human has to look at. ProcessLifecycleManager consults it for the budget-exhaustion park too, so running out of tries during a quota drain no longer produces the “re-authenticate an account” instruction.

For a quota park the sweep’s evidence is the pool itself: QuotaResetCheckerJob restores the accounts first and wakes the priority sessions blocked on them in the same pass, and only for a runtime that has an available account again, so a session is never woken into a pool that is still empty.

An auth park gets the same fast path on different evidence. “An account is available” is the whole story for a quota park — the pool was empty and now is not. It is no evidence at all for an auth park, which is what park_reason_for_pool answers whenever the pool does have something available and the runtime rejected it anyway (and as the fallback when the pool is empty or unreadable). For the common case that predicate is true by construction at park time, so waking on it alone would resume the session into the identical failure every 15 minutes.

What an auth park waits for instead is the pool’s credentials changing. park! records auth_outage_pool_fingerprint — a digest of every available account’s id and stored oauth_config, HMAC’d with the app secret because the fingerprint lands in session metadata that agents can read back — and the sweep resumes the session only once that stops describing the pool. An account added, removed, restored to active, or re-authenticated moves the digest; a rotation stamp, a quota-hit counter, or a sync that adopts an identical config does not, which is why it is content-addressed rather than an updated_at comparison.

It is a coarse signal, not a repair detector, and the code says so. The same digest also moves when RefreshRuntimeAuthTokensJob’s five-minute sweep rotates the current account’s access token on schedule — which says nothing about a parked session’s identity problem. So the fingerprint decides whether there is anything new to try, and a budget decides how often one session may act on it: MAX_EARLY_WAKES (3) per EARLY_WAKE_WINDOW (6 h). The window is what keeps the bound meaningful: without one, a pool whose credentials churn — a token sync, an account added and removed — would re-wake the same broken identity indefinitely.

The budget lives in auth_outage_early_wakes — a list of wake timestamps, pruned to the window on every write, and the one auth_outage_* key deliberately kept out of STALE_RETRY_METADATA_KEYS. It has to outlive the resume it paid for, or a re-park would hand the session a fresh budget and the cap would bound nothing. It is charged inside resume_parked!’s transaction, under the same row lock as the resume: charging afterwards would race the job that resume enqueues for the metadata column, and a failed charge would silently un-bound the cap. Past the budget — and for a park with no recorded fingerprint at all — the session stays parked until a human resumes it or the pool changes again.

Logging in from the UI

The “Authenticate” button drives a PTY-screen-scraping flow:

RuntimeLoginAttempt is being used as a cross-container message bus: the web process writes pasted_code, the worker process reads it. That read must bypass ActiveRecord’s per-request query cache, which would otherwise hide the write — so exactly one method owns it, RuntimeLoginAttempt.bus_state, and every reader goes through that. It is still a table doing a message bus’s job. Tracked in #111.

Re-authenticating the account that is live

capture! writes the DB row and stops, and the next session to spawn reads the new token out of that row. Re-auth is scratch login → validate → write the DB → done, for the current account exactly as for any other.

That used to be a trap, and it is worth remembering why. With a shared credentials file, writing only the DB made a successful re-auth of the current account a no-op: the live file every session read kept the broken tokens, the UI reported “authenticated”, and every transcript kept saying Not logged in · Please run /login. Both re-authentications during the 2026-08-22 incident only worked because the pool had already been switched away from the broken account first. There is no second store now, so there is nothing for the update to fail to reach.

The same history explains why a Claude account’s card has no Re-activate button. It existed to rewrite a live credentials file from the DB copy, and with no file to rewrite it would be a control that does nothing. A Codex account’s card keeps it: ~/.codex/auth.json is real, and re-activating the current Codex account rewrites it without recording a rotation event, because a rotation from an account to itself is not one.

Admission to the pool no longer requires a refresh round trip, either. validate_switchable still insists a refresh token exists — a pair without one is a dead end in eight hours however well its access token works right now — but it then asks the cheaper, more direct question first: does Anthropic still honour the stored access token? That is exactly what “can this account serve a session” means, and unlike a refresh it does not spend the single-use refresh token to find out. Gating on the round trip is what let the failure block its own repair: an account freshly authenticated through the UI holds a working access token and an unused refresh token, and demanding a refresh burned the latter to learn nothing about the former.

The screen-scrape is only as stable as the CLI’s output

The Claude CLI self-updates on the worker — ClaudeCodeUpdateJob runs claude update daily — so its output can change with no Zimmer deploy, and when it does, the flow breaks in production while every driver test stays green, because those tests assert against a captured buffer. (Codex is pinned in the image and moves only on a deploy, which makes the same drift visible in a diff.) That is not hypothetical: Claude Code 2.1.232 began rendering its authorization link as an OSC 8 hyperlink, which prints the URL twice — once as the escape sequence’s target, once as the visible label — and the old \S+ tail ran straight through the BEL terminator into the second copy. The panel showed a 907-character link no browser could open, so the login could not be completed at all.

Three properties keep RuntimeLoginDriver#strip_ansi and the URL patterns honest against that class of drift:

• OSC 8 hyperlinks are unwrapped to their target, not deleted. Terminals conventionally shorten a hyperlink’s visible label, so a stripper that dropped the whole sequence would be betting on that label still being the full URL.
• A URL match ends at the first control character (URL_CHAR, not \S). Whatever decoration a future CLI wraps the link in, the match cannot swallow it.
• A half-arrived escape sequence at the end of the buffer is discarded. The job re-parses a growing buffer every tick and surfaces the first URL it sees exactly once, so a chunk read that cuts a hyperlink in half would otherwise pin the panel to a truncated link for the whole attempt.

This is the concrete shape of the screen-scraping hazard recorded in Limitations.

When output does drift again, capture it from the CLI on the worker — spawn it under a PTY into a throwaway CLAUDE_CONFIG_DIR, exactly as the job does — and build the fixture from those bytes. A fixture written from memory re-tests the parser against the output it already handles.

Every attempt reaches a terminal state

The panel polls until the attempt is terminal, so an attempt that stops moving is a spinner that never stops. A worker killed hard enough to skip Ruby — deploy SIGKILL, crash, container replacement — runs no ensure and no rescue, so nothing in the job will ever write that row again. Three independent deadlines cover it, in order of how fast they fire:

Signal	Fires after	Enforced by
heartbeat_at goes stale	HEARTBEAT_TIMEOUT (90s)	login_status on the next 2s poll; CleanupRuntimeLoginAttemptsJob every 5 min
the CLI’s own lifetime	MAX_DURATION (12 min from spawn)	RuntimeLoginJob’s loop
expires_at elapses	DEFAULT_TTL (14 min from creation)	login_status and the cleanup job

RuntimeLoginJob stamps heartbeat_at every 15 seconds while it holds the CLI open; RuntimeLoginAttempt#fail_orphaned! is the single place that converts a missed deadline into a terminal status and a message naming what was missed. An elapsed window is expired; a dead worker is failed.

The heartbeat is the only one of these that survives a container restart intact. A recorded pid is meaningless once PIDs have been renumbered in a fresh container — it may be absent (reaping a live login) or reused by an unrelated process (never reaping a dead one).

Two client-side backstops close the loop, because a server that resolves an attempt correctly still leaves a frozen panel if the browser never hears about it. The poller repaints the panel with an explanation when it gives up after 10 consecutive failed polls, or when it passes the attempt’s expires_at; it never merely stops its timer, which would leave the last frame it rendered on screen looking like work still in progress.

Those 10 attempts back off — 2s, 4s, 8s, 16s, then 30s each — so the budget spans about three minutes instead of the twenty seconds a flat 2s cadence spent it in. A deploy, a lid closing, or a wifi handover is shorter than that, and a login that would have completed is no longer abandoned over one. The backoff never schedules past expires_at, so the deadline message stays as prompt as it was. And login_status answers a poll it cannot render a panel for — the row pruned after its retention window, or its account deleted mid-login, leaving the attempt detached — with a terminal panel rather than a 404, which reads to the poller as a network blip. RuntimeLoginJob treats the same case as a terminal failure: there is no account left to capture credentials into.

The login flow is screen-scraping a TUI

The command string (claude auth login --claudeai), the authorize-URL host regex, and the literal prompt /Paste code here/i are all hardcoded. ClaudeLoginDriver even hardcodes /home/rails/.local/bin/claude as its first binary candidate. If Claude Code changes any of these, login breaks wholesale.

Codex is the same story, with codex login --device-auth and a device-code regex (/\b([A-Z0-9]{4}-[A-Z0-9]{4,8})\b/) tuned to an observed 4–5 character split.