Deep ingest performance: concurrent describes + batched marks to fit a full run inside one token lifetime #12

describe-batch.mjs runs a strictly sequential describe → doc → mark loop, and each mark is a separate manifest.mjs child process that reads, parses, mutates, serialises and atomically rewrites the entire manifest. On a production tenant with ~5,000 indexed assets this makes a full deep ingest roughly a two-hour job.

That matters more than it sounds, because of a CLI defect the plugin already works around: the token stops working at roughly half its printed lifetime. gs-admin whoami keeps reporting it valid while every API call fails with Token expired and silent refresh failed. The plugin's token pre-flight rule (skills/setup/SKILL.md, Phase 1) accounts for it — usable life is remaining − 1800s of a 3600s token — which in practice leaves a working window of about 30 minutes. Reproduced at CLI 1.0.7 through 1.0.9, and on two independent tenants.

So a full ingest cannot complete inside one token today. It must survive four or five re-authentications, each one a chance for a batch to die mid-domain.

The goal of this issue is to bring a full deep ingest under one token lifetime, which removes the re-authentication problem by construction rather than managing it.

Measurements

Taken on a live run, Windows, manifest at 2.3 MB / 5,076 inventory entries. Per-asset timings are derived from last_verified inter-arrival gaps in the manifest; process costs are wall-clock over three runs.

So marking is ~12% of wall clock on the fast domains, and the describe round-trip is the other ~88%. Node startup alone — buying nothing — is 69 ms × ~5,000 assets, about 6 minutes per full ingest.

Separately, a 2.3 MB manifest is parsed and re-serialised once per asset: roughly 5,000 full parse/stringify cycles per run. The OS page cache absorbs much of the read side, so the real cost is CPU on parse and serialise plus the write and rename rather than raw disk throughput. It also grows with the inventory, since each mark rewrites a file that every previous mark enlarged.

Three changes, in order of payoff

1. Concurrent describes. The dominant cost, and the only one that changes the token maths. A bounded worker pool over the describe calls at N=6 would take the fast domains from ~1.35 s/asset to roughly 0.25 s effective — a full ingest in the ballpark of 20 minutes, inside a single token.

2. Mark in-process rather than spawning. describe-batch.mjs shells out to manifest.mjs per asset. Removing the spawn removes ~69 ms per asset of pure launch overhead.

3. Batch the marks. manifest.mjs mark --keys-file <json-array-of-keys> already accepts a batch, so the interface exists. Flushing every N assets divides the manifest-rewrite cost by N.

The constraint a fix has to respect

Per-asset marking is a deliberate durability choice, not an oversight — describe-batch.mjs:10:

marks each entry as its doc lands — so an interruption never loses more than one asset

Batching trades that away, and with a ~30-minute usable token, interruptions are routine rather than exceptional. So batching needs a recovery story before it is safe.

The natural one: docs are written to disk before the mark, so the doc files are the recoverable record. A reconcile-on-resume step — scan <slug>/<domain>/*.md against the inventory and mark anything already written — makes batched marking safe and, as a bonus, makes a mid-batch token death cost nothing at all. That step is worth having on its own merits, independent of concurrency.

Design constraints

• Keep a single writer. The manifest is one JSON file per tenant shared by every domain, save() is an atomic temp+rename, and there is no locking anywhere in scripts/ — no flock, .lock, O_EXCL or wx. Atomic prevents a torn file; it does not prevent lost updates, so two concurrent processes doing read-modify-write would silently drop marks. Concurrency belongs inside one process that owns the manifest, never across processes or agents.
• Preserve the ordering guarantee that makes a run resumable: doc written, then entry marked.
• --budget / --all semantics must not change.

Open unknowns a contributor should settle first

• API rate limits are unmeasured. Nothing is known about concurrent-request tolerance. Start at N=4 with backoff on 429 and make N a flag; a throttling storm mid-ingest is worse than a slow serial run.
• Whether concurrent describes share auth cleanly. All requests use one token from the OS keychain, and simultaneous refresh behaviour has not been examined.
• The 69 ms startup figure is Windows; other platforms will differ, though the ranking of the three changes should not.

Verifying a change

Compare last_verified inter-arrival gaps before and after on the same domain — the same method that produced the table above. A correct implementation leaves the resulting manifest equivalent to a serial run's for the same inputs (key ordering aside), with no entry left at metadata whose doc exists on disk.