chore(ci): check environment-variables.md against src

.github/workflows/ci.yml

@@ -417,6 +417,28 @@ jobs:
       - name: Check the dtype-key checker rejects a shrunk or moved scope
         run: bash scripts/ci/check_kernel_dtype_keys_test.sh
 
+  env-var-docs:
+    name: env var docs
+    # Deliberately not behind the `changes` filter, like `crate-versions` and
+    # `kernel-dtype-keys`: it needs no toolchain and runs in seconds, and it has
+    # to fire on both halves of a break. A new `MLXCEL_*` read under `src/`
+    # breaks it, and so does a docs-only change that deletes the row for a name
+    # still read; a `rust`-filtered job would miss the second. Every `MLXCEL_*`
+    # string literal in `src/**/*.rs` must be on docs/environment-variables.md
+    # or carry a reasoned entry in the script's ALLOWLIST (issue #1140). The
+    # second step is the checker's own negative coverage on synthetic trees.
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+      - name: Check every MLXCEL_* name in src is documented or allowlisted
+        run: python3 scripts/ci/check_env_var_docs.py
+      - name: Check the env-var docs checker rejects an undocumented name
+        run: python3 -m unittest tests/test_check_env_var_docs.py -v
+
   kernel-port-dispatch:
     name: kernel port dispatch
     # Not behind the `changes` filter, for the same reasons as the job above: no

CONTRIBUTING.md

@@ -47,9 +47,9 @@ Thank you for your interest in contributing to mlxcel! This document covers the
    cargo test --workspace --profile test-fast --features metal,accelerate --no-fail-fast -- --test-threads=1   # NOT gated at PR time; yours to run
    cargo deny check             # gated at PR time (advisories + licenses + sources)
    ```
-   PR-time CI in [`ci.yml`](.github/workflows/ci.yml) runs far more than `cargo fmt`, `cargo deny`, and a crate-version check: a `changes` job path-filters most of the rest by what the PR touches, and `crate-versions`, `kernel-dtype-keys`, `binary-assets`, `cuda-arch-lists`, `license-headers`, `python-tooling`, `llama-compat-manifest`, and `webui-contract` need no toolchain, run in seconds, and gate every PR unconditionally. `cargo fmt` and `cargo deny` gate on Rust-touching PRs, as before, and so does clippy again: after #21/#23 dropped it for cost, #1285 re-added it as the `clippy` job (`cargo clippy -p mlxcel --lib --tests -- -D warnings`, default features) on the shared self-hosted `GB10` runner, where a persistent target directory keeps a warm run to 27-40 seconds; it fires whenever a Rust file changed, or the PR carries the `ci:full` label. That gated slice is narrower than the full `--workspace --all-targets --features metal,accelerate` command above, and the general unit suite stays unenforced (no PR job runs `cargo test --workspace`), so run both commands above yourself before you push. `GB10` also runs further path-gated jobs, each of which the `ci:full` label forces as well: `xla-compile` on Rust changes, `xla-link` on the IREE link-recipe paths only, `cuda-sm70-compile` and `cuda-blockfloat` on the CUDA-architecture paths (the `src/lib/mlxcelverse/` tree minus its ROCm and Metal subtrees, plus the MLX build scripts) rather than on ordinary Rust changes, and `webui-installed-artifact` on Rust or WebUI changes; the hosted `webui-bundle` job gates on the WebUI path filter alone (which includes the server and CLI sources), and the hosted `mlx-pin` job on the pinned-MLX-commit paths. `rocm-build` is parked on a `rocm-gfx1151` self-hosted runner until that runner is registered and `vars.ROCM_CI_ENABLED` is set; `rocm-ci-status` reports that parked state as an advisory warning, never a failing check. `workflow-lint` (`actionlint`) and the advisory `self-hosted-gate-advisory` both fire on workflow-file changes, the latter noting the path-gated `GB10`/`rocm-gfx1151` jobs above were not re-run unless the PR carries `ci:full`. `cross-repo-refs` runs on every PR but only advises (`STRICT=1` gates it locally). Outside `ci.yml`, [`pipeline-parallel-ci.yml`](.github/workflows/pipeline-parallel-ci.yml) still runs its own path-filtered clippy and a `distributed::`-scoped `cargo test` on pipeline-parallel changes. [`nightly-verify.yml`](.github/workflows/nightly-verify.yml) runs the full `make verify` once a day on `self-hosted-macos-26-arm64` and files an issue when `main` goes red or the run does not finish, a backstop and not a substitute. The exhaustive CUDA and ROCm test suites run in no PR job; `make verify-test-cuda` and `make verify-rocm` remain their only gates, run locally.
+   PR-time CI in [`ci.yml`](.github/workflows/ci.yml) runs far more than `cargo fmt`, `cargo deny`, and a crate-version check: a `changes` job path-filters most of the rest by what the PR touches, and `crate-versions`, `kernel-dtype-keys`, `env-var-docs`, `binary-assets`, `cuda-arch-lists`, `license-headers`, `python-tooling`, `llama-compat-manifest`, and `webui-contract` need no toolchain, run in seconds, and gate every PR unconditionally. `cargo fmt` and `cargo deny` gate on Rust-touching PRs, as before, and so does clippy again: after #21/#23 dropped it for cost, #1285 re-added it as the `clippy` job (`cargo clippy -p mlxcel --lib --tests -- -D warnings`, default features) on the shared self-hosted `GB10` runner, where a persistent target directory keeps a warm run to 27-40 seconds; it fires whenever a Rust file changed, or the PR carries the `ci:full` label. That gated slice is narrower than the full `--workspace --all-targets --features metal,accelerate` command above, and the general unit suite stays unenforced (no PR job runs `cargo test --workspace`), so run both commands above yourself before you push. `GB10` also runs further path-gated jobs, each of which the `ci:full` label forces as well: `xla-compile` on Rust changes, `xla-link` on the IREE link-recipe paths only, `cuda-sm70-compile` and `cuda-blockfloat` on the CUDA-architecture paths (the `src/lib/mlxcelverse/` tree minus its ROCm and Metal subtrees, plus the MLX build scripts) rather than on ordinary Rust changes, and `webui-installed-artifact` on Rust or WebUI changes; the hosted `webui-bundle` job gates on the WebUI path filter alone (which includes the server and CLI sources), and the hosted `mlx-pin` job on the pinned-MLX-commit paths. `rocm-build` is parked on a `rocm-gfx1151` self-hosted runner until that runner is registered and `vars.ROCM_CI_ENABLED` is set; `rocm-ci-status` reports that parked state as an advisory warning, never a failing check. `workflow-lint` (`actionlint`) and the advisory `self-hosted-gate-advisory` both fire on workflow-file changes, the latter noting the path-gated `GB10`/`rocm-gfx1151` jobs above were not re-run unless the PR carries `ci:full`. `cross-repo-refs` runs on every PR but only advises (`STRICT=1` gates it locally). Outside `ci.yml`, [`pipeline-parallel-ci.yml`](.github/workflows/pipeline-parallel-ci.yml) still runs its own path-filtered clippy and a `distributed::`-scoped `cargo test` on pipeline-parallel changes. [`nightly-verify.yml`](.github/workflows/nightly-verify.yml) runs the full `make verify` once a day on `self-hosted-macos-26-arm64` and files an issue when `main` goes red or the run does not finish, a backstop and not a substitute. The exhaustive CUDA and ROCm test suites run in no PR job; `make verify-test-cuda` and `make verify-rocm` remain their only gates, run locally.
 
-   `make verify` runs nine prerequisite targets: `verify-versions`, `verify-kernel-dtype-keys`, `verify-kernel-port-dispatch`, `verify-llama-compat`, `verify-rocm-overlay`, `verify-python-tooling`, `verify-fmt`, `verify-clippy`, and `verify-test`. The last three are the fmt, clippy, and test commands above; `cargo deny check`, the fourth command in that block, is not one of them, and PR CI is what gates it. The first six are cheap consistency gates: `verify-versions` asserts every version-tracking workspace crate carries the root `mlxcel` version, `verify-kernel-dtype-keys` asserts every CUDA and HIP JIT kernel launch keys its cache on the input dtypes and that the set of launcher files has not shrunk unnoticed (#1053, #1054, #1875), `verify-kernel-port-dispatch` asserts every fused-kernel launcher chooses its port through `select_kernel_port` (#1801, #1885), and that only HIP ports with no lane-level operation are marked to run on wave64 ROCm devices (#2147), `verify-llama-compat` validates the checked-in llama-server b10621 compatibility manifest under `compat/llama-server/b10621/` structurally, with no network access required (#1443), and `verify-rocm-overlay` checks, offline, that the ROCm overlay's records agree with the MLX pin and the tree, so a pin bump that skips the ROCm retarget fails on any host (#1813), and `verify-python-tooling` runs the `tests/test_*.py` unittest suites for the bench, GPU guard, quality gate and header scripts (#2150; the GPU guard tests skip off Linux). The nightly invokes those same Makefile targets, so the local gate and CI cannot drift apart. Tests build under `[profile.test-fast]` rather than `--release`: `opt-level = 3` is kept so MLX numerics stay representative, while cross-crate LTO and the single codegen unit that make full `--release` test links expensive are dropped. `make test-fast` / `make test-fast-cuda` are the same profile with `--test-threads=1` and a `FILTER` hook for narrowing the run while you iterate. Reach for `cargo test --release --features metal,accelerate` by hand only when you suspect a defect specific to release codegen. See [`docs/installation.md`](docs/installation.md#fast-iteration-builds) for the measured comparison.
+   `make verify` runs ten prerequisite targets: `verify-versions`, `verify-kernel-dtype-keys`, `verify-env-var-docs`, `verify-kernel-port-dispatch`, `verify-llama-compat`, `verify-rocm-overlay`, `verify-python-tooling`, `verify-fmt`, `verify-clippy`, and `verify-test`. The last three are the fmt, clippy, and test commands above; `cargo deny check`, the fourth command in that block, is not one of them, and PR CI is what gates it. The first seven are cheap consistency gates: `verify-versions` asserts every version-tracking workspace crate carries the root `mlxcel` version, `verify-kernel-dtype-keys` asserts every CUDA and HIP JIT kernel launch keys its cache on the input dtypes and that the set of launcher files has not shrunk unnoticed (#1053, #1054, #1875), `verify-env-var-docs` asserts every `MLXCEL_*` name in `src/**/*.rs` is documented on [`docs/environment-variables.md`](docs/environment-variables.md) or allowlisted with a reason in `scripts/ci/check_env_var_docs.py` (#1140), `verify-kernel-port-dispatch` asserts every fused-kernel launcher chooses its port through `select_kernel_port` (#1801, #1885), and that only HIP ports with no lane-level operation are marked to run on wave64 ROCm devices (#2147), `verify-llama-compat` validates the checked-in llama-server b10621 compatibility manifest under `compat/llama-server/b10621/` structurally, with no network access required (#1443), and `verify-rocm-overlay` checks, offline, that the ROCm overlay's records agree with the MLX pin and the tree, so a pin bump that skips the ROCm retarget fails on any host (#1813), and `verify-python-tooling` runs the `tests/test_*.py` unittest suites for the bench, GPU guard, quality gate and header scripts (#2150; the GPU guard tests skip off Linux). The nightly invokes those same Makefile targets, so the local gate and CI cannot drift apart. Tests build under `[profile.test-fast]` rather than `--release`: `opt-level = 3` is kept so MLX numerics stay representative, while cross-crate LTO and the single codegen unit that make full `--release` test links expensive are dropped. `make test-fast` / `make test-fast-cuda` are the same profile with `--test-threads=1` and a `FILTER` hook for narrowing the run while you iterate. Reach for `cargo test --release --features metal,accelerate` by hand only when you suspect a defect specific to release codegen. See [`docs/installation.md`](docs/installation.md#fast-iteration-builds) for the measured comparison.
 
    The plain `make test`, `make test-verbose`, and `make test-lib` targets stay on Cargo's default `dev` profile and now export `RUST_MIN_STACK=16777216` for you. That stack bump is there for `server::reasoning_effort_tests`: its repeated renders of the pinned 8,952-byte Qwen3.8 chat template can overflow libtest's default 2 MiB per-test thread stack under unoptimized `dev`, even though the same suite passes under `test-fast` and `release`. If you bypass the Makefile and run the `dev`-profile server tests directly, keep the same prefix yourself: `RUST_MIN_STACK=16777216 cargo test --lib --features metal,accelerate server::`. `make verify-test` is intentionally unaffected because it already builds under `[profile.test-fast]`.
 

Makefile

@@ -149,7 +149,7 @@ help: ## Show this help message
 	@grep -E '^[a-zA-Z0-9_.-]+:.*?## .*$$' $(MAKEFILE_LIST) | grep -E '(clean|fmt|install)' | grep -v '^bench' | awk 'BEGIN {FS = ":.*?## "}; {printf "  $(CYAN)%-20s$(RESET) %s\n", $$1, $$2}'
 	@echo ""
 	@echo "$(BOLD)Maintenance Targets:$(RESET)"
-	@grep -E '^[a-zA-Z0-9_.-]+:.*?## .*$$' $(MAKEFILE_LIST) | grep -E '^(update|tree|outdated|verify-versions|verify-kernel-dtype-keys|verify-binary-assets|bump-version):' | awk 'BEGIN {FS = ":.*?## "}; {printf "  $(CYAN)%-20s$(RESET) %s\n", $$1, $$2}'
+	@grep -E '^[a-zA-Z0-9_.-]+:.*?## .*$$' $(MAKEFILE_LIST) | grep -E '^(update|tree|outdated|verify-versions|verify-kernel-dtype-keys|verify-env-var-docs|verify-binary-assets|bump-version):' | awk 'BEGIN {FS = ":.*?## "}; {printf "  $(CYAN)%-20s$(RESET) %s\n", $$1, $$2}'
 	@echo ""
 	@echo "$(BOLD)Variables:$(RESET)"
 	@echo "  $(CYAN)MODEL$(RESET)    Path to model directory (default: ./models/default)"
@@ -821,7 +821,7 @@ verify-rocm-smoke: ## ROCm smoke: build, link and generate on the GPU, asserting
 # "does a real forward pass still run on this device" answer even while the
 # suite is red.
 .PHONY: verify-rocm
-verify-rocm: verify-versions verify-kernel-dtype-keys verify-kernel-port-dispatch verify-llama-compat verify-rocm-overlay verify-python-tooling verify-fmt verify-clippy-rocm verify-rocm-smoke verify-test-rocm ## Run the ROCm gate locally on an AMD host (issue #1811)
+verify-rocm: verify-versions verify-kernel-dtype-keys verify-env-var-docs verify-kernel-port-dispatch verify-llama-compat verify-rocm-overlay verify-python-tooling verify-fmt verify-clippy-rocm verify-rocm-smoke verify-test-rocm ## Run the ROCm gate locally on an AMD host (issue #1811)
 	@echo "$(GREEN)[verify-rocm] OK$(RESET)"
 
 # The gate for a shared gfx1151 host (issue #2244). Build everything the gate
@@ -855,6 +855,18 @@ verify-kernel-dtype-keys: ## Assert every CUDA and HIP JIT kernel launch keys it
 	@python3 scripts/ci/check_kernel_dtype_keys.py
 	@bash scripts/ci/check_kernel_dtype_keys_test.sh
 
+# Every MLXCEL_* string literal under src/**/*.rs must appear on
+# docs/environment-variables.md or carry a reasoned ALLOWLIST entry in the
+# script (issue #1140). tests/test_check_env_var_docs.py is its negative
+# coverage: synthetic trees with a new undocumented call site, a deleted row,
+# and stale allowlist entries, each of which must fail. That suite also runs in
+# `verify-python-tooling`; it is repeated here so this target is complete alone.
+.PHONY: verify-env-var-docs
+verify-env-var-docs: ## Assert every MLXCEL_* name in src is documented in docs/environment-variables.md or allowlisted (issue #1140)
+	@echo "$(CYAN)[verify] MLXCEL_* environment variable docs...$(RESET)"
+	@python3 scripts/ci/check_env_var_docs.py
+	@python3 -m unittest tests/test_check_env_var_docs.py
+
 # check_kernel_port_dispatch_test.sh is the negative coverage for the ROCm
 # wavefront rules (#2147): it marks a shuffle-based table any-wave, adds a
 # shuffle to a marked body and calls the wave-size test seam from production
@@ -1039,7 +1051,7 @@ bump-version: ## Release: set every version-tracking crate to VERSION and sync C
 	@$(MAKE) --no-print-directory verify-versions
 
 .PHONY: verify
-verify: verify-versions verify-kernel-dtype-keys verify-kernel-port-dispatch verify-llama-compat verify-rocm-overlay verify-python-tooling verify-fmt verify-clippy verify-test ## Run the full CI-faithful gate locally (recommended before push)
+verify: verify-versions verify-kernel-dtype-keys verify-env-var-docs verify-kernel-port-dispatch verify-llama-compat verify-rocm-overlay verify-python-tooling verify-fmt verify-clippy verify-test ## Run the full CI-faithful gate locally (recommended before push)
 	@echo "$(GREEN)[verify] OK: matches the nightly-verify GitHub Actions job$(RESET)"
 
 .PHONY: verify-clean