Chart Drift has never run: MEMEX_DEPLOY_REPO_TOKEN authenticates as nothing (HTTP 401) — re-mint it

Chart Drift (.github/workflows/chart-drift.yml, daily 15 6 * * *) has failed every run since it was added — 3 for 3:

run	when	result
32001510403	2026-08-17 06:24Z	failure
31931303015	2026-08-16 06:23Z	failure
31869293070	2026-08-15 06:22Z	failure

This is the designed behaviour, not a defect in the workflow. Its header says so explicitly, and AGENTS.md → "A gate NEVER tests its own inputs" is why: a drift check that quietly skips when a credential is missing would report "no drift" on no evidence — strictly worse than not existing, because it also tells you the cluster is fine. So preflight fails RED and names what to provision. Filing this so the red is tracked rather than normalised into background noise — drift detection for memex, memex-cloud and client-i is currently not running at all.

What the run actually says

##[error]chart-drift cannot run — provision the following, then re-run:
  • secrets.MEMEX_DEPLOY_REPO_TOKEN  read-only PAT for Systemorph/Memex, where the per-env
    values.<env>.public.yaml overlays live (they must be committed there first, secret-free)

AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_SUBSCRIPTION_ID are all present. MEMEX_DEPLOY_REPO_TOKEN is the only empty one.

To make it green — three things, in this order

1. Commit the secret-free half of each env's helm overlay to Systemorph/Memex.

The workflow checks out Systemorph/Memex at deploy-repo/ and reads
deployments/aks/<ns>/values.<release>.public.yaml for each of:

namespace	release	expected file
`memex`	`memex`	`deployments/aks/memex/values.memex.public.yaml`
`memex-cloud`	`memexcloud`	`deployments/aks/memex-cloud/values.memexcloud.public.yaml`
`client-i`	`client-i`	`deployments/aks/client-i/values.client-i.public.yaml`

Include everything under config:, persistence:, keda:, probes:, ingress: — nothing under secrets:. The check compares the ConfigMap and the pod spec; it never reads a secret value. check-chart-drift.sh deliberately refuses to render with chart defaults (comparing the cluster against a chart nobody deployed produces confident nonsense), which is why the overlays have to exist rather than being optional. Optional per-env portal-patch.json is picked up as --expect-patch if present.

2. Add MEMEX_DEPLOY_REPO_TOKEN as a repo secret — a fine-grained PAT scoped to Systemorph/Memex with Contents: Read-only and nothing else.

3. Grant the CI identity AKS command access. The app registration behind secrets.AZURE_CLIENT_ID currently only has ACR rights (that is all main-cd.yml needs). Chart Drift additionally needs Microsoft.ContainerService/managedClusters/runCommand/action on the cluster. Scoped to the cluster resource memexaks-cluster (RG memex-aks-rg):

Azure Kubernetes Service Cluster User Role
Azure Kubernetes Service RBAC Reader

The check only ever issues kubectl get, so read-only is sufficient.

Why it matters

check-chart-drift.sh was written 2026-08-13 to detect exactly the class of failure that took memex.meshweaver.cloud down the following day — and it had no caller at all until this workflow. Every day it stays unprovisioned is a day cluster-vs-chart divergence goes undetected. Related: the hand-applied config that helm upgrade silently reverts is precisely what this catches.

Verifying

After all three land: gh workflow run chart-drift.yml --ref main. A completed run reports drift, if any, as red with each divergence classified CLUSTER-ONLY / CHART-ONLY / DIFFERS; green means the three namespaces match their charts.

Narrowed to item 3 — and it is one manual action away from green.

The workflow's own header records the state: .github/workflows/chart-drift.yml marks items 1 and 2 [DONE 2026-08-17], and item 3 [OUTSTANDING] at :52 — secrets.MEMEX_DEPLOY_REPO_TOKEN has never been provisioned.

Consequence, measured: the workflow has been running RED daily ever since, through 2026-08-25 (most recent runs/32816762564). So "Chart Drift has never run" remains literally true, and it is now also a standing red that trains people to ignore it.

Remaining work is a maintainer action, not a code change: mint the token and add it as MEMEX_DEPLOY_REPO_TOKEN.

Worth noting the gate is correctly built — it fails RED rather than skipping, which is why this is visible at all. Compare #2249, filed tonight: the same preflight pattern in the satellites fails red for a missing Dependabot secret, which is exactly the behaviour you want.

Correcting this issue: the secret is provisioned. It has never been valid.

The title and step 2 of this issue are wrong, and have been for eight days. Measured, not inferred:

$ gh api repos/Systemorph/MeshWeaver/actions/secrets/MEMEX_DEPLOY_REPO_TOKEN
{"name":"MEMEX_DEPLOY_REPO_TOKEN",
 "created_at":"2026-08-17T11:12:42Z",
 "updated_at":"2026-08-17T11:12:42Z"}     ← created_at == updated_at: never re-set since

And the preflight log of the most recent run (32816762564, 2026-08-25) — note it gets past the presence check, so AZURE_* and the token are all non-empty:

All external inputs are present.
##[error]secrets.MEMEX_DEPLOY_REPO_TOKEN is set but GitHub rejects it (HTTP 401, Bad credentials).

Every run since the secret was stored has ended there: the workflow_dispatch eleven seconds after it landed (32023855740, 11:12:53Z), then nine consecutive schedules 08-18 → 08-25. Twelve failures total, and the first three (08-15/16/17-06:24, quoted in this issue) are the only ones that were the originally-diagnosed "unprovisioned".

So the remaining action is RE-MINT, not provision

401 rules out the two adjacent explanations by construction, which is why the probe distinguishes them:

code	means	ruled out?
401	the value does not authenticate as anything	← this is what we get
403	valid token, org owner has not approved it	yes, we'd see 403
404	valid token, repo not in its scope / no Contents:Read	yes, we'd see 404

What is left is a value that is not a token: revoked, expired, truncated on copy, or never the token at all — a clipboard-piped gh secret set stores whatever was on the clipboard. A trailing newline does this too (gh secret set NAME < token.txt), goes straight into the Authorization header, and is invisible in the Actions UI because every secret renders as ***.

Status of the three steps

step	actual state
1. overlays committed to `Systemorph/Memex`	✅ done 2026-08-17 (`2cd3eee`) — header item 2
2. `MEMEX_DEPLOY_REPO_TOKEN`	⚠️ provisioned 2026-08-17, rejected 401 ever since — needs a re-mint
3. AKS `runCommand` rights	✅ already held; the item was written from an assumption that turned out to be wrong (the app registration has Cluster Admin Role)

Do not read the daily red as "1 and 3 are outstanding too" — preflight never gets far enough to exercise them.

What #2278 changes, and what it does not

Nothing about the workflow's shape: the preflight is correct as designed and stays exactly as strict. Two things:

The header stops lying. Item 3 said [OUTSTANDING], which reads as "not provisioned" and is what kept this misdiagnosed.
The 401 branch now says what is actually stored — a length, a prefix class (github_pat_ / ghp_ / gho_-family / none), and a whitespace check. All derived; the value itself is never printed and never narrowed. Verified firing on a real trailing newline (38 characters where the clean token is 37, plus the named printf %s | gh secret set fix).

Chart Drift stays RED until someone re-mints the PAT — that is one manual action, and it is the whole remaining scope of this issue. It will not go green on a workflow edit, and the first completing run is expected to be legitimately very red (a local run found 121 divergences across 149 compared fields — the accumulated kubectl patch backlog, which is the finding, not a defect in the gate).

Re-titling suggestion: "Chart Drift: MEMEX_DEPLOY_REPO_TOKEN authenticates as nothing (HTTP 401) — re-mint it".

Merge pull request #2309 from Systemorph/fix/1709-app-token-cross-repo

Merge pull request #2278 from Systemorph/fix/gates-that-never-fired

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-08-17)

Worked through this. Two of the three requirements are now done, and one of them was wrong in the original write-up. Only the PAT is left.

✅ 1. Per-env overlays — committed

Systemorph/Memex@2cd3eee adds all three:

deployments/aks/memex/values.memex.public.yaml
deployments/aks/client-i/values.client-i.public.yaml
deployments/aks/memex-cloud/values.memexcloud.public.yaml

Captured from the live releases with helm get values <release> -n <ns> -o yaml, so they are the values helm actually last deployed, not an idealised copy.

🚨 Correction to this issue's instructions — "nothing under secrets:" is NOT a sufficient rule.
pgbackrest.azure.accountKey is a live Azure storage account key and it sits outside the secrets: block (memex release). A deny-list overlay built to the original instruction would have committed it to git permanently.

So the files are built as an allow-list: only config, ingress, persistence, image, replicas, resources are copied; parameters: and pgbackrest: are dropped wholesale. Verified by keyword scan and a high-entropy scan — the sole secret-shaped match is ingress.secretName (a Kubernetes object name), and the remaining long values are OAuth ClientIds and a TenantId, which are public identifiers whose matching client secrets stay in the excluded block.

✅ 2. AKS command access — already granted; this issue was wrong

The app registration behind secrets.AZURE_CLIENT_ID is github-actions-deploy (appId fb51b7e9-6554-4857-80d7-23608d8347fc), and it already holds Azure Kubernetes Service Cluster Admin Role scoped to memexaks-cluster — which includes Microsoft.ContainerService/managedClusters/runCommand/action.

No role assignment is needed. The claim that it "only has ACR rights" was an assumption in the workflow header, not a checked fact.

⏳ 3. `MEMEX_DEPLOY_REPO_TOKEN` — the only remaining step, and it needs a human

A fine-grained PAT cannot be minted through the API or CLI, so this one is genuinely yours:

Scope it to Systemorph/Memex, permission Contents: Read-only, nothing else.
Add it as repo secret MEMEX_DEPLOY_REPO_TOKEN on Systemorph/MeshWeaver.

Deliberately not worked around. Reusing the existing broad GH_PAT would widen what the drift job can reach for no good reason, and moving the overlays into the public repo would publish the deployment inventory — an information-disclosure decision that isn't mine to make, and one the private-repo convention exists to prevent.

The gate works — verified, and it already found a lot

Rather than assume, I ran check-chart-drift.sh locally against the real cluster with the new overlay:

149 fields compared — 121 divergence(s)

So the plumbing is proven end-to-end. That number is genuine accumulated drift, not a bug in the overlay, and it needs triage. A sample of what it caught:

finding	detail
`DIFFERS ConfigMap Authentication__Microsoft__ClientId`	chart `CHANGE_ME_aad_application_client_id` vs live `eed68df1-…` — a `helm upgrade` today would overwrite working auth with the placeholder
`DIFFERS ConfigMap AzureFoundry__Endpoint`, `ModelTier__*`, `Features__Ai__Providers__Anthropic`	the DeepSeek/AzureFoundry routing is hand-applied; the chart has never heard of it
`DIFFERS spec.replicas`	chart 2 vs live 1
`DIFFERS livenessProbe` / `readinessProbe` / `startupProbe`	live carries `initialDelaySeconds` and `failureThreshold: 1080` that the chart does not
`DIFFERS lifecycle.preStop`	live is missing the `command -v curl` guard the chart renders

This is the concrete, measured form of the thing we already knew qualitatively — the portals are configured by kubectl patch, and helm upgrade reverts it. Each row is a "decide which side is authoritative, then make the other match" decision.

Once the PAT lands, the scheduled run will report this same list daily until it is triaged — which is the point, but worth knowing before it starts paging.

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-08-21)

The premise has moved twice — correcting the record

1. The token IS provisioned, and the blocker is different

secrets.MEMEX_DEPLOY_REPO_TOKEN was added on 2026-08-17T11:12:42Z. This issue's title still says "provision" it, which is no longer the ask.

What the preflight actually reports, on today's run (32454219730) and every run since 08-17:

##[error] secrets.MEMEX_DEPLOY_REPO_TOKEN is set but GitHub rejects it (HTTP 401, Bad credentials).

So it is set but invalid — revoked, expired, or truncated on copy. Worth noting the preflight is behaving exactly as designed: #1774 taught it to assert the token is VALID, not merely present, precisely so this state could not read as green. It is working.

2. …but the maintainer's direction is not to re-mint it at all

"there is such a token in memex.systemorph.com. all deploys should happen through there"

That is correct, and verified:

memex.systemorph.com holds a Deployments space that GitSyncs from Systemorph/Memex — the private deploy repo — subdirectory mesh/Deployments, lastSyncedAt: 2026-08-19T20:06:37Z. It is reading that repo successfully right now, with a credential that works.
That space is already the deployment control plane: Hosting/Deployment records for memex, memex-cloud and client-i, plus Hosting/BackupStore and Hosting/InstanceAction.

So the instance that chart-drift needs data from already has the credential and the synced content. Minting a second PAT into GitHub Actions secrets duplicates a credential that exists, and re-creates the class of problem #1835/#1844 is retiring elsewhere:

a consumer that needed one package asked for a standing GitHub credential to a whole repository, because there was no smaller thing to ask for.

Chart-drift needs the per-env values.<env>.public.yaml overlays. Those are in Systemorph/Memex, which the Deployments space already syncs.

Proposed re-scope

Retitle to "Chart Drift reads chart state from the Deployments space, not from a repo PAT." The work becomes:

Chart-drift obtains overlay/chart state from memex.systemorph.com's Deployments space rather than checking out Systemorph/Memex.
MEMEX_DEPLOY_REPO_TOKEN is then removed, not re-minted — one fewer standing credential, and no expiry to rediscover in three months.
The preflight keeps asserting whatever the new path needs, with the same fail-RED-naming-it contract.

Two things unchanged and still required either way: the AKS runCommand rights (the header notes the existing app registration already holds them, no role assignment needed), and the secret-free per-env overlays being committed to the deploy repo first.

Open question for the maintainer: whether chart-drift reaches the Deployments space over the MCP/HTTP surface, or whether the drift check should move into the mesh entirely and become a Hosting/ capability beside Hosting/Deployment — which would fit the instance-lifecycle work already landing there.

@rbuergi

(re-posted 2026-10-11; originally by @rbuergi on 2026-08-21)

Re-checked today — two of the three are DONE, and the remaining blocker is not the one this issue names

Still red, now 8 runs for 8 (latest 32454219730, 2026-08-21 06:23Z). But the reason has changed since this was filed, and the issue text now sends you to the wrong place.

#	prerequisite	state today
1	secret-free overlays committed to `Systemorph/Memex`	✅ done — all three files exist
2	`MEMEX_DEPLOY_REPO_TOKEN` repo secret	⚠️ set, but the value is rejected
3	AKS `runCommand` / RBAC Reader for the CI identity	❓ unknown — never reached

1 is done

gh api repos/Systemorph/Memex/contents/<path> resolves all three:

deployments/aks/memex/values.memex.public.yaml          → present
deployments/aks/memex-cloud/values.memexcloud.public.yaml → present
deployments/aks/client-i/values.client-i.public.yaml          → present

So check-chart-drift.sh has the overlays it refuses to run without.

2 is the live blocker — and it is a bad VALUE, not a missing secret

The secret exists (gh secret list → MEMEX_DEPLOY_REPO_TOKEN, created 2026-08-17T11:12:42Z), so "MEMEX_DEPLOY_REPO_TOKEN is the only empty one" is no longer the diagnosis. What the preflight actually says:

##[error]secrets.MEMEX_DEPLOY_REPO_TOKEN is set but GitHub rejects it (HTTP 401, Bad credentials).
  The stored value is not a usable token. Most often it is not the token at all —
  a clipboard-piped 'gh secret set' stores whatever was on the clipboard — or the
  token was revoked, expired, or truncated on copy. Re-mint at
  https://github.com/settings/personal-access-tokens and store it again.

The timeline is worth noting: the secret was created at 11:12:42Z on 2026-08-17 and a manual run fired 11 seconds later at 11:12:53Z — and failed. It has failed on 401 every day since. So this has been "provisioned but broken" for four days while reading, from the outside, exactly like the "never provisioned" state this issue describes.

The fix is one step: re-mint the fine-grained PAT (resource owner Systemorph, repository Systemorph/Memex, Contents: Read-only, nothing else) and store it again. That is a credential operation on this repository, so I have not done it — flagging it for someone who can.

401 is specifically not the org-approval case: the preflight distinguishes them, and a fine-grained PAT still pending owner approval returns 403, while one scoped to the wrong repo returns 404. 401 means the stored bytes are not a valid token at all.

3 is still unverified, and will stay that way until 2 is fixed

The preflight fails closed at the token probe, so the run never reaches the cluster and nothing has yet exercised Microsoft.ContainerService/managedClusters/runCommand/action. Expect a second red after fixing the token — that is progress, not a regression. The RBAC grant in step 3 of this issue should be applied at the same time rather than discovered a day later.

Credit where it is due

The preflight has clearly been improved since this issue was written — it now separates 401 / 403 / 404 / 000 / 5xx and prints the correct remedy for each, and fails closed on the indeterminate ones ("a gate that cannot reach its subject has not checked it"). That is the only reason this update could say which of the three prerequisites is broken instead of just "still red".

Why this still matters, concretely

I ran into the payoff on #1788 an hour ago. All three prod namespaces carry memex-migration-deployment-… pods, while the chart on main templates only a kind: Job and has never had a deployment.yaml in its history. That is a textbook CLUSTER-ONLY divergence — exactly one of the three verdicts this check reports — and it was found by a human reading pod lists during an incident, four days after the fact. Every day Chart Drift stays red is a day that class of divergence has no detector.

Navigation Menu

Repository navigation

Chart Drift has never run: MEMEX_DEPLOY_REPO_TOKEN authenticates as nothing (HTTP 401) — re-mint it #1709

What the run actually says

To make it green — three things, in this order

Why it matters

Verifying

Correcting this issue: the secret is provisioned. It has never been valid.

So the remaining action is RE-MINT, not provision

Status of the three steps

What #2278 changes, and what it does not

✅ 1. Per-env overlays — committed

✅ 2. AKS command access — already granted; this issue was wrong

⏳ 3. `MEMEX_DEPLOY_REPO_TOKEN` — the only remaining step, and it needs a human

The gate works — verified, and it already found a lot

The premise has moved twice — correcting the record

1. The token IS provisioned, and the blocker is different

2. …but the maintainer's direction is not to re-mint it at all

Proposed re-scope

Re-checked today — two of the three are DONE, and the remaining blocker is not the one this issue names

1 is done

2 is the live blocker — and it is a bad VALUE, not a missing secret

3 is still unverified, and will stay that way until 2 is fixed

Credit where it is due

Why this still matters, concretely

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Chart Drift has never run: MEMEX_DEPLOY_REPO_TOKEN authenticates as nothing (HTTP 401) — re-mint it #1709

Description

What the run actually says

To make it green — three things, in this order

Why it matters

Verifying

Activity

rbuergi commented on Aug 25, 2026

rbuergi commented on Aug 25, 2026

Correcting this issue: the secret is provisioned. It has never been valid.

So the remaining action is RE-MINT, not provision

Status of the three steps

What #2278 changes, and what it does not

rbuergi commented on Oct 11, 2026

✅ 1. Per-env overlays — committed

✅ 2. AKS command access — already granted; this issue was wrong

⏳ 3. MEMEX_DEPLOY_REPO_TOKEN — the only remaining step, and it needs a human

The gate works — verified, and it already found a lot

rbuergi commented on Oct 11, 2026

The premise has moved twice — correcting the record

1. The token IS provisioned, and the blocker is different

2. …but the maintainer's direction is not to re-mint it at all

Proposed re-scope

rbuergi commented on Oct 11, 2026

Re-checked today — two of the three are DONE, and the remaining blocker is not the one this issue names

1 is done

2 is the live blocker — and it is a bad VALUE, not a missing secret

3 is still unverified, and will stay that way until 2 is fixed

Credit where it is due

Why this still matters, concretely

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

⏳ 3. `MEMEX_DEPLOY_REPO_TOKEN` — the only remaining step, and it needs a human