How Zimmer consumes AIR

Zimmer touches AIR in exactly two places: a read path that asks “what artifacts exist?” and a write path that says “prepare this directory.”

The catalog is self-contained and offline

Zimmer’s air.json declares a catalog named zimmer-catalog with no catalogs field and no github:// URIs — only six local index paths, gitProtocol: "https", and two extensions (@pulsemcp/air-adapter-claude, @pulsemcp/air-secrets-env).

Everything lands under @local/, so no short id is contested and every artifact is addressed by its bare short id — see Artifact names are qualified, and reduced to a token for what changes when a deployment composes a second catalog.

The catalog’s own description states the intent: “resolves fully offline (no private GitHub catalogs, no network), so the app’s config services always resolve non-empty data.”

What’s in it: 10 skills — 7 Zimmer-specific ones (category: zimmer) plus 3 vendored generic workflow skills (category: workflow: open-pr, wait-for-ci, recover-from-compaction-thrashing) — 22 MCP servers, 11 roots, 4 plugins, 1 hook, 5 references. The zimmer root turns 9 of those skills on by default and exactly one MCP server, playwright-custom; awaken-waiting-sessions and zimmer-fleet default on fleet-maintenance, and zimmer-sessions on dashboard-reprioritizer. It also defaults to both ci-workflow and screenshots-videos, and a plugin’s own bundle rides along with it — so a zimmer session additionally receives the git-push-ci-reminder hook and the remote-fs-screenshots server, neither of which is named in any default_in_roots.

Those counts are asserted against a live resolve by test/docs/zimmer_integration_catalog_counts_test.rb, so adding a catalog entry without updating this paragraph fails CI rather than leaving the page quietly stale.

Vendored generic skills are adapted, not mirrored

A generic skill vendored here is the current revision of that skill with its links made to resolve against this catalog. The two reasons a link cannot come across verbatim are the same two the self-contained property implies:

• The reference is not in this catalog. open-pr upstream deep-links four references Zimmer does not carry, and wait-for-ci deep-links one of those same four (HARNESS_BASH_DENIALS.md). All four live in a private repo and some of them describe the private production deployment; tadasant/zimmer is public, so copying them in would leak deployment detail and break the offline-resolve promise. The substance those passages needed is written inline in the vendored body instead, and the links are gone. Incident citations that name a private repo get the same treatment: the mechanism stays, the repo name and the issue number go.
• The anchor is not in the vendored reference. A references/GIT_WORKFLOW.md#some-heading link resolves only if this catalog’s copy of GIT_WORKFLOW.md has that heading, and the reference drifts behind the skill just as easily as the skill drifts behind upstream. The fix is to re-vendor the missing section — adapted the same way a skill is, so the deployment-specific material and private cross-links do not come with it — and only to drop the pointer when the upstream prose does not belong here at all. Either way the skill states the rule in full inline; where both exist, the skill is the operative instruction and the reference carries the shared convention and the incident behind it.

Neither kind of breakage fails air resolve: the artifact resolves fine and just points at prose that is not there. The no skill links a reference or a heading the catalog does not carry test in test/services/skills_config_test.rb is what catches it — it walks every references/*.md link in every SKILL.md in the catalog and checks the file exists, the skill declares it, and the anchor is a real heading, then does the same for each skill’s in-page #anchor links, which go stale the moment a re-vendoring renames a section.

air.json vs air.production.json

They are content-identical today. The split is a seam — it lets the production image pin its own catalog sources without touching the dev/test config. Selection is per-environment: development/test use air.json, production/staging use air.production.json. AIR_CONFIG always wins.

The github-cache machinery is inert here, not gone

Because the catalog is local-only, all of AirCatalogService’s github-cache machinery (~/.air/cache/github, resolved_sha_for, pinnable_catalogs, catalog pins) does nothing on this deployment: there is no github:// URI for it to act on. It stays because AIR_CONFIG can point at a catalog that does declare them, and the @pulsemcp/air-provider-github extension that resolves them ships in every image.

What it must not do is read as live. The settings page hides its Catalog Pins card unless at least one catalog is pinnable, AIR_CATALOG_REF warns at boot when it pinned nothing, and the tests that cover the pinning path run in CI against synthetic remote-catalog fixtures rather than skipping for want of a real one (#69).

Artifact names are qualified, and reduced to a token

AIR addresses every artifact as @<scope>/<id> — local for the deployment’s own indexes, <owner>/<repo> for anything a github:// catalog contributes. The qualification is what lets two composed catalogs each contribute a slack MCP server without colliding.

Zimmer used to ask AIR to throw it away (air resolve --no-scope), which produced shortname-keyed output and hard-failed the whole resolve on any cross-scope shortname collision. One duplicated short id therefore degraded the entire catalog to last-known-good until an operator dropped one side via air.json#exclude (#208).

It now resolves qualified, and ArtifactIdentity reduces each qualified ID to a canonical token — the value Zimmer stores in sessions.mcp_servers / catalog_skills / metadata["agent_root_key"], validates against the catalog, shows in a picker, and hands back to air prepare:

Situation	Canonical token
One catalog contributes this short id	the bare short id (slack)
Several do, one of them @local	@local keeps the bare short id; the others are @owner/repo/slack
Several do, none @local	every one is @owner/repo/slack; no bare token exists

@local keeps the bare token because a bare name written into Zimmer’s own database or index files most plausibly means the deployment’s own artifact — the same “intra-catalog first” rule AIR applies to a short reference inside an artifact body. The practical consequence is the one that matters: composing a new catalog that happens to share a short id cannot change the meaning of a name already stored here.

Three forms are accepted wherever an artifact is named — the canonical token, the fully-qualified @scope/id, and a bare short id exactly one catalog contributes — so every identifier written before this existed still resolves and no re-keying migration was needed.

Two deliberate seams:

• What Zimmer stores is not always what AIR is handed. AIR resolves a bare shortname only when exactly one scope contributes it, so AirPrepareService expands every id to its qualified form before building the air prepare argv.
• Reference fields inside entries are rewritten back. A root’s default_skills, a plugin’s skills/mcp_servers/hooks, a skill’s references come back from AIR qualified; Zimmer rewrites them to canonical tokens on the way in, which is why default_in_roots and every root default still reads as bare ids.

The qualified ID and the “another catalog contributes this short id too” flag are stamped onto each entry (__zimmer_qualified_id, __zimmer_contested) so they survive the CatalogSnapshot round-trip. An older snapshot carries neither, and serves exactly as it always did.

A dangling reference is treated as a failed resolve

This is the sharpest coupling between the two systems, and the most brittle thing in Zimmer.

AIR exits 0 when it drops an unresolvable reference, printing a warning to stderr. So Zimmer scans stderr for two literal strings ("references unknown" and "Dropping the reference") and, if both appear, raises CatalogError despite the exit code being 0.

Why so aggressive? Because a dropped reference is exactly what strips a root’s default_skills, default_mcp_servers, and default_hooks. Persisting that tree would misconfigure every session created against it — and overwrite the good snapshot with degraded data. So a degraded resolve never reaches persist_snapshot.

The two-marker test exists because references dropped intentionally by air.json#exclude share the second marker but are expected.

This is a string copy of another project’s log output

app/services/air_catalog_service.rb says it plainly:

“Matching on AIR’s exact stderr wording — a string copy, not a stable contract — is brittle, but AIR exposes no machine-readable signal for dropped references.”

If AIR ever rewords that warning, Zimmer will silently start accepting degraded catalogs and misconfiguring sessions. There is no test that would catch it, because the test suite uses the same AIR version. Tracked in #66.

The blast radius is the entire test suite

test/test_helper.rb pre-warms the catalog at boot, before parallelize forks its workers. So a catalog that fails to resolve does not fail one test — it fails every test that creates a session, all at once, with ActiveRecord::RecordInvalid.

A single dangling reference (a plugin bundling a skill that no longer exists, a default_in_roots naming an unknown root) reddens the whole suite. CONTRIBUTING.md says it: if you see a sudden wave of RecordInvalid across unrelated session tests, suspect the catalog before you suspect your change.

A missing body is quieter than a dangling reference

The stderr marker check only sees references between entries. It does not see a registered artifact whose path has nothing behind it — that resolves clean and is silently skipped later, at air prepare, when the adapter tries to copy a directory that isn’t there.

git-push-ci-reminder was exactly that for a while (#65): registered in hooks/hooks.json, bundled by plugins/ci-workflow, default_in_roots: ["zimmer"], and no hooks/git-push-ci-reminder/ on disk. The body exists now, and SkillsConfig/HooksConfig’s tests assert every registered artifact has one — but AIR itself still won’t tell you.

An empty resolve is a failed resolve too

The stderr check above only speaks up about references between entries AIR found. Point AIR at a config whose index paths do not exist and it finds no index files at all: nothing to drop, nothing printed, exit 0, and a tree that is legitimately and completely empty.

That is the quieter failure, and it used to be the silent one — degraded? stayed false, and the empty tree was persisted as the new last-known-good, overwriting the good one. So reject_empty_resolve! raises CatalogError when a resolve comes back with no artifacts of any of the six types, which routes it down the same path as a dropped reference: serve the previous catalog, flag degraded?, and never reach persist_snapshot.

A tree with entries for one type and nothing for the other five is not empty — a lopsided catalog is a normal catalog. Only nothing-at-all trips this.

With no last-known-good to fall back on, the CatalogError surfaces the way every other resolve failure does: each of the six façades rescues it to an empty value and the session form renders resolve_failure as a banner. The pickers are still empty — a catalog this empty cannot create a session either way — but now they come with a stated reason. That includes each façade’s raw config reader, which /settings reads through DeploymentInfoService: those used to call entries_for unrescued, so a catalog that resolved to nothing would have 500’d the one page an operator would go to in order to clear the offending pin.

This is a second detector, not a replacement for the stderr one, and it does not close #66: a reworded AIR warning would still let a partially dropped catalog through. It was added with #1078, where a relocated config’s relative index paths produced exactly this shape of empty tree and nothing noticed.

The snapshot is the source of truth

In production the web (Puma) and worker (GoodJob) run in separate containers, so AIR’s ~/.air/cache/github provider clones are per-container. Only a process that has just fetched can resolve a fresh tree. So resolving is a write, and serving is a read:

• Resolve (write). refresh! runs air update, then air resolve, and stores the result as the newest CatalogSnapshot row in Postgres. It runs once at boot in every process (so a new image’s in-repo catalog is live as soon as that image boots), on the worker’s */15 CatalogRefreshJob cron, behind the Refresh catalogs button (which enqueues that same job on the worker), and when a catalog pin is saved. A process with no snapshot at all resolves on first use.
• Serve (read). A 60-second in-memory TTL on the parsed tree (CATALOG_CACHE_TTL). When it expires, the process reads the newest snapshot’s header (id, resolved_at, failure columns: one narrow query) and loads the tree only if that row is newer than the one it is serving. No process re-resolves on a timer, and there is no background thread anywhere.

So the web serves whatever the worker last resolved, at most a minute after the worker stored it.

The snapshot carries what a process used to read off its own disk:

Fact	Column	Read by
When the writer last fetched its clones (FETCH_HEAD mtimes)	fetched_at	last_refreshed_at, the “Updated X ago” label
The commit each pinnable catalog resolved to, for HEAD and its pinned ref	catalog_shas	resolved_sha_for, the settings page’s catalog pins
Whether the latest refresh attempt failed, and its scrubbed error	failed_at, failure_message	degraded?, resolve_failure

Two things still read the local disk, on purpose. repo_root_for wants a clone to read files out of, which a snapshot cannot carry; its only caller, WarmSkillsCacheJob, runs on the worker. air prepare materializes skills from the local cache too, and a fork or unarchive runs it on the web container. So before every prepare, AirPrepareService asks AirCatalogService.disk_cache_behind_snapshot? and runs an air update when this container’s clones were fetched before the snapshot’s, bounded by CATALOG_CATCH_UP_TIMEOUT_SECONDS (60s) — much shorter than air prepare’s own cap, because this one runs inside a web request. On the worker the check is a directory glob and nothing more.

Last-known-good and degraded

A failed refresh (a failed air update or a failed air resolve) stores nothing new. It writes its error, scrubbed of credentials, onto the newest snapshot as failed_at / failure_message — unless that row was resolved after the failed attempt began, in which case another process succeeded while this one was failing and the row it wrote is fresh, so it is left alone. Every process that serves that row reports degraded? and resolve_failure from it on its next TTL tick, so the session form’s failure banner and get_configs tell the truth in a web process that never resolves. The next successful refresh stores a fresh row, which clears it everywhere.

The process whose refresh failed alerts: it logs at error on its own healthy→degraded transition (or CatalogRefreshJob does) and at info after that. A process that learns of the failure from the snapshot logs at info only, so one failure pages once, not once per process.

Within the process that resolves, load! falls back in order: in-memory tree → CatalogSnapshot.latest → re-raise. Only a first-ever cold boot with a broken catalog and no snapshot raises.

The write path: AirPrepareService

Invoked synchronously from AgentSessionJob on waiting → running:

air prepare <adapter> \
  --target <clone> \
  --no-subagent-merge \
  --without-defaults \
  [--root <name>] \
  --skill <id>...  --mcp-server <id>...  --hook <id>...  --plugin <id>...

Two decisions here are load-bearing:

--without-defaults is deliberate. Zimmer already stores the final resolved per-session artifact lists in the database — the UI’s PATCH endpoints mutate them directly. AIR 0.0.30 flipped --skill semantics from “replace defaults” to “add to defaults.” Without --without-defaults, a user removing a default artifact in the UI would watch AIR silently re-add it from the root defaults. So Zimmer uses AIR’s root-defaults machinery at read time (to seed a new session) and explicitly bypasses it at prepare time.

Secrets flow through the environment. SecretsLoader.all is merged into the subprocess env; @pulsemcp/air-secrets-env substitutes the ${VAR} placeholders into .mcp.json; AIR then fails the prepare if any ${VAR} survived, which Zimmer catches as a graceful, non-paging SecretResolutionError.

Resilience

The whole invocation runs under BoundedSubprocess with a hard wall-clock timeout (it SIGKILLs the process group), with retry-and-backoff on transient failures. There’s a special case for “Root not found”: it triggers one inline bounded air update (cache bust) and a retry — because a freshly-merged root can legitimately be absent from a worker’s up-to-15-minutes-stale cache. If it’s still absent, it raises a graceful RootResolutionError.

An unparseable JSON file is retried on that same ladder. air-sdk JSON.parses the files the adapter wrote into the target — .mcp.json and .claude/settings.json — without a guard, and air-core does the same for air.json and the catalog indexes, so a failure there exits 1 with a bare Node parse error and no path.

For the two files in the target directory, that is only reachable as a race. Neither can be a file that was already broken on disk, because the Claude adapter rescues its own parse failure and rewrites both from scratch before the SDK reads them. It takes a second writer changing one between the adapter’s write and that later read — and air prepare re-runs on every follow-up, resume and unarchive, over a directory a previous job for the same session may still be tearing down. A malformed air.json or catalog index reaches the same signature and is not a race; retrying costs one bounded ladder and the failure stays loud either way.

Because AIR names no file, AirPrepareService prepends what the target’s config files actually looked like when it failed. In the race that description reports everything as parsing, which is itself the confirmation; a file genuinely broken on disk shows up as UNPARSEABLE and is a different bug. The enrichment is skipped when AIR’s message already carries a path of its own.

Requested artifact ids — MCP servers, skills, hooks and plugins — get one more guard, before the invocation. Each column is validated against the catalog when the session is created, but the catalog moves on independently: an artifact renamed (pr → open-pr) or removed (gmail-tadas412-readonly) leaves a stale id in a long-lived session’s stored config. air prepare hard-rejects an unknown id with exit 1 (Error: Unknown MCP server ID "gmail-tadas412-readonly". …), which would fail every later resume, unarchive and follow-up of that session. So AirPrepareService#reconciled_catalog_selection drops any id not in the live catalog and prepares with the survivors. Catalog drift is expected, not a fault, so it logs at WARN and pages nobody. It records the dropped ids in the session’s custom_metadata["dropped_unknown_catalog_ids"], keyed by column, and AgentSessionJob tells the agent in a <dropped-catalog-artifacts> block on every later prompt, so it can pick a successor or say it cannot do the task without one. This mirrors Trigger#heal_catalog_references! (which heals the trigger path) and gives an unknown artifact the same non-fatal degradation an unknown root already gets. The pruned list is persisted (update_column, so no validation runs): a session does not prepare once — every resume, unarchive, and mid-run clone recreation re-runs air prepare, so an in-memory-only scrub would re-discover the same stale id on every turn.

A fork (including the status-summary fork) gets the same treatment one step earlier. It is a new row, so Session validation judges every id it copies, and a stale one would fail create!. ForkSessionService copies only the ids the catalog still knows, carries the source’s dropped_unknown_catalog_ids record over, and adds to it anything it dropped itself.

This is the drift path only. A fresh start_session naming an unknown id is a caller error and is still rejected at validation time, before any session exists to prepare.

Two guards keep that write from doing damage. If the catalog failed to load and left the catalog facade empty (so every id would look stale), the list is left untouched rather than stripped. And nothing is persisted while the catalog is degraded — a failed resolve usually does not empty the facade, it serves a last-known-good tree, which is non-empty and can predate a rename, so an id that is perfectly valid today looks stale against it. Dropping such an id in memory costs one prepare; writing that drop back would erase a valid id permanently and undo the backfill that repointed it. A failed write likewise degrades to a warning, so the scrub can still keep the prepare alive.

Renaming a skill in the catalog is what creates those stale ids in the first place, so a rename should ship with a data backfill that repoints existing catalog_skills rows (sessions and triggers) from the old id to the new one — the heal alone only drops the id, which silently strips the skill from long-lived sessions and from every session a trigger spawns. db/migrate/20260801120000_backfill_renamed_open_pr_skill_id.rb is the worked example for the pr → open-pr rename. Keep such a migration pinned to the one known rename: the catalog is a runtime dependency that can resolve differently at migration time, so pruning against it is not deterministic. General staleness stays the runtime heal’s job.

The AIR CLI is installed lazily, at runtime

AirPrepareService.ensure_air_installed! runs npm install into AIR_INSTALL_DIR on first use, pinned to AIR_CLI_VERSION = "0.13.0" — the CLI plus both adapters, the secrets-env transform, and the GitHub provider. Guarded by a version marker file, a binary health check (air --version), and a cross-process install lock.

The install builds into a .incoming staging directory and swaps the finished tree into place. The published directory therefore stays readable for the whole install, which matters because readers hold no lock: AirCatalogService spawns <AIR_INSTALL_DIR>/node_modules/.bin/air directly, so an install that emptied the directory first would hand every concurrent reader Errno::ENOENT for the minute npm takes. A failed install leaves the previous tree in place.

AIR_INSTALL_DIR defaults to /opt/air-cli when that path exists or /opt is writable, and to ~/.cache/air-cli otherwise — but only in a deployed environment. Development resolves to ~/.cache/air-cli and test to ~/.cache/air-cli-test, whatever is on disk; an explicit AIR_INSTALL_DIR still outranks all of it. Agent sessions run on the same host as the production container, and both test/test_helper.rb (at suite boot) and the catalog initializer under bin/agent-dev call ensure_air_installed!, so a session running tests or a dev server in its clone would otherwise reinstall over the directory the live app is shelling out to. That is not hypothetical: on 2026-09-01 a branch that added an adapter to the package set did exactly this and produced 10 production ERROR records in a minute, including an ActionView::Template::Error rendering the dashboard’s session cards.

Two versions to keep in lockstep

Dockerfile.base bakes @pulsemcp/[email protected] into /opt/air-cli and touches a .air-version-0.13.0 marker; AirPrepareService::AIR_CLI_VERSION is what ensure_air_installed! looks for. If they drift, the image’s pre-baked CLI is ignored and the first session on a fresh container replaces it with a different version — on the session’s launch path.

test/contracts/air_config_parity_test.rb asserts they agree (and that air.json and air.production.json still declare the same catalog), so drift fails a test rather than a deploy. Bump both together.

The six façades

SkillsConfig, AgentRootsConfig, ServersConfig, PluginsConfig, HooksConfig, and ReferencesConfig are thin read-models over AirCatalogService.entries_for(:type). Each shapes raw resolve output into a Ruby value object, and each swallows CatalogError into an empty array with a warning — so a catalog failure degrades the UI instead of returning a 500.

That degrade is deliberate, but on its own it is indistinguishable from a fresh install: every picker on the session form renders empty and nothing says why (#112). AirCatalogService.resolve_failure closes that gap. It records the last failed resolve — including the case degraded? cannot see, where no last-known-good tree exists, load! re-raises, and the façades rescue to [] — and the session form renders it as a banner saying whether the lists below are empty or merely stale, with air resolve’s own error text — scrubbed of every credential this process holds (#319), because /sessions/new has no Rails-layer authentication and the process that produced that text is handed AIR_GITHUB_TOKEN.

The agent side reads through the same façades, so it had the same blind spot: get_configs would report “No MCP servers available” and start_session would happily build a session against a catalog that never resolved. Mcp::Tools::GetConfigs now prepends the same fact — empty versus stale, plus when the failure was seen.

Not the same fidelity, deliberately. The banner prints air resolve’s error text, and that process is handed AIR_GITHUB_TOKEN, so even scrubbed it is not something to echo onto an agent channel — the scrub only knows credentials Zimmer itself holds. What an agent needs in order not to act wrongly is the fact and its age; the text stays with the operator, on the form and in the logs.

Never parse the index files directly. That’s the rule in AGENTS.md and it’s a good one: the indexes are AIR’s input; the resolved tree is Zimmer’s data model. The resolved tree is what Zimmer consumes, and it differs from the raw index (references canonicalized, default_in_roots inverted and deleted, paths absolutized).