A frozen opt-in cursor lets stale stars reverse a completed opt-out #9

Context

DevIndex's opt-in cursor (optin-sync.json) has not advanced since 2026-03-29T09:58:10Z — commit fee7327841, 143 days before filing. optout-sync.json has never advanced at all; it still reads {"lastCheck": null}.

That is not a stuck file. It is the visible symptom of the working set's read side and write side naming different artifacts, and the consequence is that a completed opt-out can be silently reversed and the person re-published.

Found while verifying the DevIndex extraction (#17375). Observation and inference are separated below: the mechanism is verified from code and live probes; the population of affected people is explicitly not measured, because enumerating it means handling exactly the personal data this ticket is about.

Live latest-open sweep: checked the latest 20 open issues at 2026-08-19T17:56Z; no equivalent found. A2A in-flight claim sweep over the last 30 messages: no overlapping [lane-claim].

The Problem

The DevIndex pipeline hydrates its working set from config.publishedWorkingSet.baseUrl, which resolves to neomjs.com/node_modules/neo.mjs/apps/devindex/resources/data/ — the copy served from neomjs/pages. Since neomjs/neo#17375, the pipeline publishes to a content-plane prefix named by the DEVINDEX_PUBLISH_BUCKET repository variable. Those are different artifacts. Each run adopts the pages copy, publishes its own working set where nothing serves it, and the next run starts from the pages copy again.

Neo's own publish step never restored the cursors either: .github/workflows/data-sync-pipeline.yml:283-287 stages users.jsonl and the portal data, and nothing else. So the cursors have been frozen in the committed tree since March and reset on every run since.

Why a frozen cursor is not merely wasteful. OptIn.collectStargazerOptIns() pages the opt-in repository's stargazers newest-first and stops at lastCheck:

if (lastCheck && starredAt <= lastCheck) { stopFetching = true; break }

With lastCheck pinned to March, every star placed since then is re-collected on every run — and stargazers are one of the two signals that reverse a blocklist entry (OptIn.mjs, "Remove from blocklist if they are there (ONLY stargazers and self-issues)"). So an opt-in star that predates a later opt-out is replayed indefinitely, removing that person from the blocklist again on every run.

And the un-block reaches the published index. Stage order is opt-in → opt-out → spider → updater. Manager.mjs runs Cleanup.run() as pre-run hygiene for spider and update only — optin and optout run none. By the time Cleanup performs blocklist enforcement it is reading the blocklist.json that OptIn has already rewritten, so the person is no longer blocklisted, is not pruned, and is enriched into users.jsonl and published.

The opt-out is not repaired on a later run either: OptOut re-adds only people it can still see. Somebody who opted out via the issue template had that issue closed after processing, and somebody who opted out by starring is invited by our own guides to remove the star afterwards. Neither is visible to the next run.

The Architectural Reality

Paths are in neomjs/devindex; the workflow reference under "The Problem" is neomjs/neo.

Measured, not inferred:

• the working-set manifest returns HTTP 404 at the read URL — pages carries no manifest, so digest verification is inert
• neomjs.com/dist/devindex/… returns HTTP 404; that host proxies to pages and the content-plane prefix is not routed
• the bucket's own public endpoint returns HTTP 403 — private, mounted into Cloud Run
• devindex run 32280343186 logged No published manifest — adopting the fetched set unverified in all three hydrating stages, then published successfully. Both halves are true at once, which is the whole defect.

The Fix

Point the read side at the artifact the pipeline publishes, so the working set round-trips through one place. config.publishedWorkingSet.baseUrl becomes the served URL of the published prefix; nothing else in Storage.hydrateWorkingSet() changes, and manifest verification becomes live the moment it can fetch a manifest.

Reaching that URL is a hosting decision, not a code choice, and it is the reason this is filed rather than fixed:

1. Route the content plane — serve the published prefix through the existing middleware, which already mounts the bucket. Needs a middleware-v2 deploy. Recommended: it keeps the artifact private-by-default and same-origin.

2. Public prefix — expose the prefix directly and point baseUrl at it. No deploy, but it adds a second public surface for the same data and moves egress off the current path.

3. Publish the working set to neomjs/devindex's own GitHub Pages, via actions/deploy-pages. The artifact path deploys straight from the workflow — no commit, no branch, so none of the history cost DevIndex collection leaves neo, and stops committing what it generates neo#17375 exists to remove. Measured against the published limits: a site may be no larger than 1 GB, bandwidth is a soft 100 GB/month, and builds a soft 10 per hour. The working set is ~26.45 MiB and the schedule is 12 runs/day, so all three have wide margin. Enabling Pages on that repository is a settings change, not a deploy.

   The honest cost, and it is why this is listed third rather than first: the dated archive/ copy must stay in object storage. A Pages deployment replaces the whole site each run, so carrying a 30-day archive would mean re-uploading it every two hours and would reach the 1 GB cap inside two months. That splits one lifecycle across two systems — working set on Pages, archive in the bucket — and a set that is "published" in two places is exactly the shape the manifest exists to make unambiguous. Option 1 keeps one destination.

   No new exposure either way: this data is already served publicly from neomjs.com. What differs between the options is the number of surfaces to reason about, not whether the data is public.

All three need the operator — a middleware-v2 deploy, a bucket ACL change, or a repository settings change. None is a code choice, and the code change is the same one line in every case.

Either unblocks the cursor: once the pipeline reads what it wrote, lastCheck advances and the replay stops.

Contract Ledger Matrix

Decision Record impact

none — verified rather than assumed. ADR 0004 governs the on-disk shape of resources/content/ (its "bucket" is an archive chunk bucket, not object storage); ADR 0014 classifies Agent OS scheduler lanes and the Brain's deployment topology. Neither governs the DevIndex published working set or the website content plane. No ADR mentions the content plane except ADR 0004 in the unrelated sense above.

Acceptance Criteria

• config.publishedWorkingSet.baseUrl resolves to the artifact publishWorkingSet.mjs writes — demonstrated by fetching the manifest from that URL and getting a document rather than a 404
• A pipeline run logs no adopting the fetched set unverified line, i.e. the manifest was fetched and every digest matched
• optin-sync.json advances across two consecutive scheduled runs: the second run's published lastCheck is strictly newer than the first's
• A spec covers the replay directly: given a blocklisted login whose opt-in star predates the cursor, OptIn does not remove it from the blocklist. The spec must fail when the cursor is pinned to a stale value — a test that cannot fail on the frozen-cursor case does not cover this
• learn/OptOut.md's statement that removal is permanent is true under the shipped behaviour, or the page is corrected
• Post-merge only: confirm on a real scheduled run, not a dispatch — the freeze is only observable across runs

Out of Scope

• The middleware-v2 deploy itself, and any change to what other routes serve
• Hosting for the DevIndex app (same decision, different artifact — it needs its own resolution)
• Reclaiming generated-artifact history — that is Reclaim the generated-artifact history in neo and pages neo#17376
• Re-designing opt-in/opt-out as a mechanism. The issue flow and the star flow are both fine; the cursor is what is broken
• Quantifying who was affected. It requires joining the blocklist against a stargazer list, which is the personal data this ticket exists to protect

Avoided Traps

• Adding a timestamp comparison so an old star cannot reverse a newer opt-out. Treats the symptom, and blocklist.json is a flat array of logins with no timestamps, so it would need a schema change to fix something that disappears when the cursor advances.
• Committing the cursors back to git. That is the publication state machine DevIndex collection leaves neo, and stops committing what it generates neo#17375 exists to delete, reintroduced for two small files — and it would restore the per-run commit that made neo's .git 4.1 GB.
• Failing closed when the manifest is absent. Every run would fail until the loop closes, which converts a silent defect into an outage without fixing it. The warning was corrected instead (devindex Storage.mjs).
• Treating the frozen cursor as a neo bug. It reads that way — the stale value lives in neo's tree — but neo never persisted it either. The freeze belongs to the publication path, not to a repository.

Related

• DevIndex collection leaves neo, and stops committing what it generates neo#17375 — DevIndex collection leaves neo (the extraction that surfaced this)
• Reclaim the generated-artifact history in neo and pages neo#17376 — Reclaim the generated-artifact history
• Register neomjs/devindex as a pull-mode tenant on the local Agent OS neo#17377 — Register neomjs/devindex as a pull-mode tenant
• feat(ci): DevIndex collection leaves the neo pipeline (#17375) neo#17391 — merge-order dependency: dev still refreshes the pages copy the pipeline reads. Merging it before this ticket closes freezes the index rather than merely leaving it stale.

Handoff Retrieval Hints

• query_raw_memories: "devindex published working set read side write side different artifacts"
• query_raw_memories: "opt-in cursor frozen stale star reverses opt-out"
• devindex commit range: 62fb114..HEAD (the extraction's read-side work, including the corrected unverified-adoption warning)
• devindex run 32280343186 — a full collection that succeeded and logged unverified adoption in the same run

Origin Session ID: 44746e37-a5f9-44c4-8c9d-f664247f0e38