feat(corpus): private corpus support, and the shared library from #11 - #12
Merged
Conversation
The four manifests the tools read sat beside the tools themselves, so scripts/ mixed executables with the data they consume. Move them under scripts/data/ and point every reader at the new path. PATTERNS_FILENAME moves with them, which is why .gitignore's comment block and the README prose describing it change in the same commit — that path is named in three places and they must not drift. _tree() in the publisher tests created root/scripts before writing the patterns file at PATTERNS_FILENAME. That only worked while the constant was one level deep; derive the directory from the constant instead.
Puts the tests in their own package instead of interleaving test_*.py with the tools under test. scripts/tests/__init__.py makes the directory a package, which is what keeps `python3 -m unittest discover -s scripts` — the command CI runs today — finding all 98, and additionally allows the more explicit `-s scripts/tests -t scripts` form. Both were verified. Deliberately no scripts/__init__.py: pytest walks up while __init__.py exists and prepends the first directory without one, so leaving scripts/ a plain directory is what puts it on sys.path and lets the tests import the tooling the way the entry points do. conftest.py makes that explicit rather than load-bearing-by-accident. That also retires the sys.path.insert in the EPUB tests, which existed only to reach a sibling module. sys.path now appears nowhere in scripts/ except conftest.py. The `if __name__ == "__main__"` block goes with it: running a test file by path puts scripts/tests/ on sys.path instead of scripts/, so it could not have worked from the new location anyway.
Brings the repo onto the polyrepo's uv/pytest/ruff/mypy convention. It was the only repo here with Python and none of it. The scripts stay stdlib-only at runtime — nothing in the dev group is ever imported by them, so the credential-free `python3 scripts/verify_corpus.py` path is untouched. The ruff config is corpus protection, not style. Unconfigured, `ruff format` reads Jupyter notebooks natively and would rewrite four fixtures under jupyter/math/ while failing to parse three more, and `ruff check --fix` would delete the two deliberately unused imports from code/hello.py — F401 is a safe fix and those imports are the entire point of that fixture. force-exclude is the line that matters: without it exclusions apply only while walking directories, so any explicitly passed path bypasses them, which is what an editor-on-save or `ruff format jupyter/` does. test_lint_scope.py asserts that behaviourally against the real binary rather than trusting the config, and cross-checks that poly.toml and pyproject.toml still describe the same corpus. Its top-level directory set comes from git, not the filesystem: only about 30 of the 53 corpus directories exist in a fresh clone, so a filesystem-derived expectation would pass vacuously in CI. Adopting uv also opened a publish hazard. corpus_paths() walks the working tree rather than asking git, the patterns are gitignore-shaped so one without '/' matches a basename at any depth, and the list contains *.png, *.pdf and *.zip — which installed packages ship as bundled assets. A .venv at the repo root was therefore reachable by a publish to the public bucket. It is now pruned from the walk, rejected by the forbidden-path guard, and gitignored, with tests that fail if any of the three is removed. BLE001 is deliberately not ignored globally, unlike upstream: doing so makes every `# noqa: BLE001 - <reason>` unused, and RUF100 then strips the reason along with the directive.
The first two shared modules, and the two smallest. paths.py replaces nine separate `Path(__file__).resolve().parent.parent` computations. That arithmetic was correct only for a file at one particular depth, and a wrong root does not raise — it just finds nothing — so moving a file broke it silently. Deriving everything from one module's own location means the depth is written down once. git_repo_root keeps asking git rather than deriving from __file__, and the publisher keeps using it. The two answers differ when a tool is run by path against a different working tree, and the publisher feeds this straight to `git -C` for its tracked-file guard, so the guard has to be asking about the same repository git is. Changing that is a separate argument, not this one. hashing.py collapses three implementations into one. Two were byte-identical chunked readers carrying their own READ_CHUNK_SIZE; the third, in build_diagram_pdfs, slurped the whole file. That is the one real behaviour change here: the digests are identical, but the corpus contains a 62 MiB object and the peak memory was being paid for nothing. test_hashing.py asserts the agreement across the chunk boundary, which is where a chunked reader actually goes wrong, rather than only on small inputs.
patterns.py puts the two pattern predicates side by side, which is the point of the module. They look alike and are not: matches_corpus_pattern answers "is this file corpus?" with gitignore semantics, where a pattern without '/' matches a basename at ANY depth, while matches_any answers "did the caller's --include select this manifest path?" with plain fnmatch. Having them in separate files read as an accidental duplicate. load_patterns also grows an optional patterns_path, so a corpus outside this repo can bring its own list. manifest.py takes the corpus.lock.json shape out of the publisher, and with it the byte-stability contract. Consumers fetch through xberg-io/actions/fetch-test-documents, which hashes the file to key its object cache, so the sort in build_manifest and the dumps/newline in write_manifest decide whether every consumer's cache survives a republish. Nothing tested that path. The publisher's tests exercise --dry-run, which deliberately never writes the manifest, and CI cannot publish at all because the corpus binaries are not in git. So the write path only ever ran on a maintainer's machine, and only where noticing was already too late. test_manifest.py closes that: it feeds the committed manifest's own objects back through both functions and requires the bytes to match. No network, no corpus binaries, no credentials. Confirmed it fails on an indent change.
The concurrency limit was configured three ways: a MAX_WORKERS constant in two tools, a --jobs flag defaulting to 8 in two more, and 6 in the fifth. Now there is one flag, one help string, and one default. fetch_corpus and verify_corpus gain --jobs, which is purely additive — the consumers that invoke these by path never pass flags, so the contract with the xberg repo's ~14 references is untouched. fetch_regression keeps 6, and the reason is now written down next to the number instead of being indistinguishable from a typo: its ~1,500 requests go to gutenberg.org, ebi.ac.uk and arxiv.org, which throttle bulk clients, while everything else here talks to one GCS bucket that does not care. The honest fix is a per-host limit rather than a global one, and the comment says so. map_parallel preserves input order because two callers report a per-item failure list and a stable order is what makes a failing run reproducible for whoever reads it; run_parallel does not, because its callers only tally.
Five call sites become one retry policy over two deliberate transports. The transports stay two on purpose. curl is the path consumers actually take: fixtures are materialised in CI by xberg-io/actions/fetch-test-documents, whose fetch.sh downloads with curl, and verify_corpus exists to prove the bucket still serves exactly that. Had it switched to urllib, CI would stay green while asserting something about a TLS stack, proxy handling and redirect policy that no consumer uses. urllib stays for the provenance fetchers, which pull from eleven third-party hosts where real exception types are worth having and 1,483 requests should not each buy a subprocess. What was accidental is now gone. Two call sites retried three times with no delay at all — the same request into the same transient failure, three times within milliseconds — and two did not retry. There is now one policy with real backoff, and fetch_corpus and verify_corpus --sample gain retries, which is the CI flake surface. The five unexplained timeouts become three values with their derivation written next to them. 300s was not arbitrary: it belongs to one URL, the govdocs1 archive that serves 267 regression members in a single request. Every other regression entry is a direct file. head_batch splits along a knowledge boundary rather than moving wholesale. curl's batching, the write-out format and its parsing are transport, so they go to CurlTransport.head_many; mapping a URL back to its pin and phrasing the failure is corpus knowledge and stays in verify_corpus. Batching is deliberately not on the Transport protocol — urllib cannot do it, and pretending otherwise would be a fake abstraction. The payoff is that the write-out parser is now unit-testable, including the case where curl exits 0 but a response never parsed, which previously read as success. materialize.py takes the third duplication issue #11 names but assigns no module. All three copies produced the same status vocabulary and the same sidecar behaviour, but fetch_regression reported `error {error}` without the type, so a bare RuntimeError printed as the word "error" and nothing else. Verified against the live bucket: 575/575 metadata, 5/5 content.
Adds the mechanism for a second corpus whose index lives outside this
repository, and the guards that make pointing the publisher at one safe.
fetch_corpus gains --manifest, --root and --auth; publish_corpus gains
--manifest, --root and --patterns. Every one defaults to today's value, so
`python3 scripts/fetch_corpus.py` and `publish_corpus.py --bucket
xberg-test-documents` are unchanged — which matters because ~14 places in
the xberg repo tell people to run the first by path, and the dry-run output
is byte-identical before and after.
Authentication is opt-in and anonymous stays the default. That is not a
convenience: this repo's CI proves the public bucket is still fetchable with
no credentials at all, and a tool that quietly authenticated would make that
proof meaningless. AdcCredential shells out to gcloud rather than importing
google.auth, because the package is stdlib-only and because WIF in CI
populates ADC exactly as a local login does — one path serves both. It
re-acquires as the token ages: WIF tokens last an hour and cannot be
extended, and a 15 GB fetch on a slow runner can outlive one, so a
credential minted once at startup would 401 halfway through.
Three guards, because publishing is not reversible and the public bucket
cannot un-serve an object:
- a non-default root or manifest aimed at the public bucket is refused
outright, before any network call;
- --root must be the manifest's own directory unless --allow-external-root
says otherwise. This is the one that catches aiming a level too high:
patterns are gitignore-shaped, so a bare `*.zip` matches a basename at
any depth and a parent directory sweeps in whatever sits beside the
corpus;
- any non-default publish prints resolved root, file count, byte total and
extension histogram, and asks. 4,412 files and 4,419 files are
distinguishable at a glance; an exit code is not.
A refusal prints its reason and exits 1 rather than showing a traceback —
it is a designed outcome, not a crash.
The tracked-file guard now runs only for this repository, where corpus
binaries are deliberately untracked. A corpus root outside a git working
tree has nothing to ask git about.
Every fixture in the new tests is a neutral placeholder. The corpus this
exists for belongs to a design partner whose filenames identify them, and
this repository is public.
…pus root upload_extra_files raises when ATTRIBUTIONS.md, LICENSES.md or ground_truth/corpus_manifest.json is missing, and it runs after the objects are uploaded. Publishing a corpus root that is not this repository would therefore push the whole corpus and only then fail, because those files are this repository's licence notices and live beside its corpus rather than inside it. A corpus elsewhere keeps its provenance with its own manifest. Caught by dry-running the private path before uploading anything.
The 80-line specification of what is stripped and what is preserved moves with the code it describes, and is now surfaced as argparse epilog so --help still prints it. The entry point gains a real parser: it previously hand-rolled argv handling and printed __doc__ on wrong argc, which meant `--help` exited non-zero and it was the one tool the smoke test could not check. All ten now answer --help.
Four reviews of the branch found real problems. The ones that mattered: strip_svg_graph_metadata lost its "nothing to strip" guard when main() was rewritten around argparse. The original refused to write and returned 1 when the output matched the input; the new one wrote unconditionally. diagrams/README.md regenerates the whole fixture set in a bare shell loop, so a producer that stopped emitting metadata would have yielded geometry fixtures byte-identical to their parents, exit 0, no stderr — a fixture pair that measures nothing, and nothing downstream would have noticed. The root-above-corpus guard was a tautology. resolve_targets derives the manifest from the root when --manifest is omitted, so manifest.parent == root by construction and the check could not fire in the exact case it existed for. It now compares the root against the directory the explicitly-named targets actually point at, and names the tighter root in the message. The public-bucket guard never looked at --patterns, though a pattern file selects the byte set as directly as the root does: --patterns with `*` in it would have swept the whole working tree into the world-readable bucket while root and manifest still looked like the defaults. is_default_root keyed off which flags were typed rather than the resolved targets, so `--root .` from the repo root — semantically identical to passing nothing — silently dropped both the tracked-corpus guard and the attribution refresh. A credential failure was being retried. It fails identically every time, so `--auth` without a valid login would have spawned three gcloud subprocesses and three seconds of backoff per object across thousands of objects before printing the same message it could have printed at the start. It is now its own non-retryable error type. AdcCredential also takes a lock — token() is called from every worker thread and the first batch stampeded gcloud — and dates the token from after acquisition rather than before, since gcloud can take a second or two and erring short is the wrong direction when the point is surviving a 15 GB transfer. Two tests could not fail, proven by mutation. The manifest serialisation test asserted the shape of the committed lock file rather than the output of write_manifest, so indent=4 with no trailing newline left it green. The fetch flag test never imported fetch_corpus at all, so making auth unconditional — the exact regression it named — passed. Transport selection is now extracted as build_transport() and the test drives it. New coverage where mutants had been surviving silently: pool.py had none at all (reversing map_parallel's order, forcing DEFAULT_JOBS to 1, and ignoring add_jobs_argument's default all passed); verify_corpus had none, though it is the only thing this repo's CI actually proves — dropping the Content-Length comparison and replacing the deterministic sample with a prefix both passed. Each new test was checked to fail against its mutant. The user-agent test now asserts the header is sent rather than that the constant exists, because removing it from the request survived the whole suite. mypy is now clean over the tooling, and scoped to it: strict mode over a test suite built on stub transports and injected runners demands annotations on every throwaway helper, which is noise. This matches xberg-io/actions, which runs mypy over its scripts and not its tests. fetch_direct now returns a list like fetch_shard, which makes the pool homogeneous and retires an isinstance check at the call site.
This was the only repo in the polyrepo with Python and no .ai-rulez/, so CONTRIBUTING.md's instruction to run `npx ai-rulez generate` produced nothing. Follows sceptre's shape: core + cicd includes from agent-conventions, python/cicd/documentation/default-commands builtins. The value is the context and rules, which record what is easy to get wrong here and impossible to infer from the code: - corpus-model — the binaries are not in git, corpus.lock.json's bytes key every consumer's fetch cache, and CI cannot publish because a checkout has nothing to upload. - tooling — why http.py keeps two transports (curl is the path consumers take, so verifying with urllib would assert something no consumer uses), why the timeouts differ, and why both test runners must stay green. - private-corpus — the flags, ADC auth, and why an anchored `data/*` pattern beats an extension list: it cannot silently under-publish. - Five critical rules: fixture bytes are load-bearing, entry-point paths are a cross-repo API, the tools import nothing but the stdlib, the manifest's bytes are the contract, and this repository is public. Two local subagents: corpus-tooling-engineer for scripts/, corpus-curator for fixtures and their provenance. Nine more come from the shared modules. poly.toml picks up the polyrepo root's rumdl disable list — ai-rulez authors agents as front-matter plus prose, so MD041's "first line must be an H1" is wrong for them by construction. Generated output (CLAUDE.md, AGENTS.md, .claude/, .codex/, .mcp.json) is gitignored via --gitignore.
The tooling targets Python 3.10+, where PEP 604 unions and PEP 585 generics evaluate natively, so the import buys nothing. Eleven files carried it and nothing checked; TID251 now bans it, which is what stops it coming back. Worth recording why this was a manual pass: UP010 removes unnecessary __future__ imports and has an always-available fix, but it deliberately leaves `annotations` alone. That one still changes runtime behaviour, since PEP 563 was never made the default — so no autofix exists and the ban has to be enforced rather than repaired. Verified all ten entry points still answer --help on 3.10, the declared floor, and the suite passes there too.
publish_corpus.py was 559 lines doing four unrelated jobs. It is now 183: its docstring, its parser, main(), and the exit. backends.py splits along the obvious seam. LocalDirBackend exists only so the upload path can be tested without a bucket or credentials, so it and the Protocol it satisfies are a natural module — and keeping a second real implementation beside the Protocol is what stops the seam rotting. publish.py takes the enumeration, the guards, staging and the uploads. The guards carry their reasoning with them: the public bucket cannot un-serve an object, and patterns match basenames at any depth, so both failure modes are silent and permanent. Verified the dry-run reports the same 538 unique objects and 3 attribution files as before, and corpus.lock.json is untouched.
…orpus fetch_corpus.py drops to 103 lines and verify_corpus.py to 56 — docstring, parser, main, exit. verify.py keeps the interpretation of a HEAD result: mapping a URL back to its pin, comparing Content-Length, phrasing the failure. Issuing the request stays in the transport. That split is why the write-out parser became unit-testable, including curl exiting 0 with a response that never parsed — the case that previously read as success. Verified live against the public bucket: 575/575 metadata, and all ten entry points still answer --help from a foreign directory.
The builder and the fetcher become library modules; the entry points keep their parsers. nested_30000 keeps its exact name, because EPUB_EDGE_CASES.md names that symbol in prose. The determinism test is the proof this moved unchanged: it rebuilds all fourteen synthesized files and compares each sha256 against the manifest pin, so a byte of drift anywhere in the builder fails it.
Moving the EPUB tools into a subpackage left `Path` and `Any` unimported in corpus_tools/epub/fetch.py and its `__file__`-relative manifest path pointing two directories too deep. ruff reported nothing, all 186 tests stayed green, and `python3 scripts/fetch_epub_edge_cases.py` raised NameError at import — the exact failure mode fourteen call sites in the xberg repo would have hit. The manifest now comes from paths.py, which is what that module exists for. So the smoke test becomes a real test rather than something to remember. test_entry_points.py runs every tool's --help in a subprocess from a temporary directory, checks each data file resolves, and walks the runtime import graph with ast to prove nothing third-party crept in. Running from elsewhere is the point: a tool that only works when the cwd happens to be the repository is broken for every consumer. Parsing rather than grepping that import graph is also the point — the first version matched a docstring line beginning "from the content instead..." and reported it as an import, and a test that cries wolf gets switched off. check_diagram_ground_truth.py gains the parser it never had, so it answers --help like the other nine, and --only to check a single fixture.
verify-corpus.yaml becomes single-purpose: the bucket check, on the runner's bare python3, with no setup step and no install. A second ~keep says so explicitly, because the missing setup-python reads as an oversight and is not one — it is the executable proof that a consumer with a stock interpreter can resolve this corpus. The unit tests move to test-unit.yml, which is allowed to install things. validate.yml wires up the poly gate that poly.toml has been describing with nothing enforcing it. Python is pinned to 3.10, the declared floor, so requires-python becomes something CI checks rather than something the manifest asserts; poly is pinned so a rate-limited "resolve latest" request cannot kill the job, and so a poly release cannot change what this gate enforces without a commit here. test-unit.yml runs pytest on 3.10 and 3.14, then re-runs the same suite through python3 -m unittest with nothing installed, because that runner is the same guarantee one level up. It is deliberately not path-filtered: most PRs here add corpus documents and touch nothing under scripts/, and a filtered workflow that never runs leaves a required check pending forever. The final `git diff --exit-code` is a tripwire for a future edit that drops a --check or adds a --fix. Taskfile.yml fixes CONTRIBUTING.md's instruction to run `task setup`, which had nothing to run. Every command is scoped, and `task format` ends by failing if anything outside scripts/ changed.
Adds a README section on serving a corpus whose index lives elsewhere: the flags, ADC auth, what the guards refuse and why, and why an anchored pattern beats an extension list. All placeholders — <private-repo>, <namespace> — because this repository is public. Three things were already wrong before this branch: - README named `scripts/normalize_gt.py`, which does not exist here. It is in the xberg repo beside `build_corpus.py`, which the same section already attributes correctly two paragraphs earlier. - CONTRIBUTING told contributors to run `git lfs install && git lfs pull`, though README says the corpus moved off LFS; the binaries are not in git at all. It now points at `task setup` and the fetcher. - CONTRIBUTING told them to run `task setup` in a repo with no Taskfile. There is one now. The test command in README and diagrams/README moves to `uv run pytest`, and the CI paragraph says why the bucket job deliberately has no setup step.
Completes the layout issue #11 asks for. Every entry point is now its docstring, its parser, main(), and the exit — the largest is 184 lines and most are under 80. diagrams/render.py takes the renderer invocations and the font check; diagrams/recipes.py takes the fixture table and build(), deliberately free of subprocess calls at import time because test_diagram_manifest imports RECIPES in CI where no renderer exists. The EMBEDDABLE_FAMILIES allowlist moves as one block with its comment: ATTRIBUTIONS.md cites it as a licence-compliance mechanism, so the reasoning has to travel with the constant. diagrams/ground_truth.py, math_binaries.py and regression.py follow the same shape. Both fetchers now take their manifest path from paths.py rather than computing it from __file__, which is what makes the move safe — a __file__-relative data path silently points somewhere else the moment its module changes depth, and finding nothing does not raise. The entry-point test caught two breaks during this: a lost ENGINES table and a missing import. That is the second time it has paid for itself.
Closes the last coverage holes from the branch review, each verified against the mutant that used to survive. fetch_corpus.main had none: only fetch_one was tested, so nothing checked what gets fetched or where it lands. Disabling the --include filter and ignoring --root both passed the whole suite. The first quietly pulls ~580 MiB into every consumer's CI; the second writes into the wrong tree. Proving that second one could fail wrote three files straight into the working tree — matched by corpus patterns, hidden by .gitignore, invisible to `git status`, and therefore something the next publish would have picked up. Removed, and the test now asserts against REPO_ROOT rather than cwd, which is where a broken --root actually lands. fetch_regression's shard path had none either, and it is the highest-risk code left: govdocs1 publishes archives, so one download yields 267 objects located by basename inside the zip. The missing-member branch, whose deletion survived, now has a test — along with the mismatch sidecar, the already-current no-op that must not re-download hundreds of megabytes, and a shard failure reporting per wanted member. The curl stubs swallowed **kwargs, so nothing checked that `fetch` gets bytes and `head_many` gets text=True. The two genuinely differ — one decodes stderr itself, the other parses stdout as a string — so flipping either flag passed the suite and would have failed against the real subprocess module. Also: resolve_objects' missing-file branch, which is what stops a manifest pinning an object the bucket will never serve; publish_corpus.parse_args, including that both safety flags default to the safe setting; and the font allowlist ATTRIBUTIONS.md cites as a licence-compliance mechanism — its parsing is split out so it can be tested without qpdf, since it otherwise only runs behind a renderer-dependent build and never executes in CI. 98 -> 215 tests.
…gs do The polyrepo type-checks with pyrefly, not mypy, and runs it through poly rather than as a dev dependency — it is a system tool like poly itself, which is why no sibling lists it in [dependency-groups]. Confirmed that `poly lint .` executes it in its whole-project phase by introducing a deliberate bad return and watching poly report it, so validate.yml already covers type-checking and test-unit.yml no longer needs a step for it. That workflow's `setup-python: true` turns out to exist for exactly this reason: pyrefly resolves imports against the interpreter it is handed. The larger fix is that poly was not configured. It does NOT read pyproject.toml's [tool.ruff] — verified: a file containing `from __future__ import annotations` was clean to `poly lint` and a TID251 error to `uv run ruff check`. The commit hook and the CI gate were enforcing different rules and nothing said so. poly.toml now carries a [lint.python.ruff] block mirroring the select, ignore, per-file-ignores and limits, matching how liter-llm, crawlberg and actions configure theirs. pyproject.toml keeps its own [tool.ruff] rather than deferring entirely, because that is what constrains a human running `ruff format .` or an editor-on-save — neither reads poly.toml, and its force-exclude is the corpus protection. Two tools, two configs, one corpus, and a test that fails when they disagree. One asymmetry is recorded rather than papered over: poly cannot express `banned-api`. It is a ruff setting rather than a rule code, and poly accepts `flake8_tidy_imports_banned_api` without complaint and without effect — so the future-import ban is enforced by the ruff step in test-unit.yml and not by poly. A test asserts that, so the next person finds it written down.
The poly gate failed on the runner with `pyrefly: not found`. My previous commit message asserted pyrefly is a system tool absent from every sibling's dependency groups — that was wrong, and the CI failure is what proved it. liter-llm and crawlberg both carry `pyrefly>=1.1.1` in their dev group, and they have to: poly runs pyrefly in its whole-project phase, and reusable-validate.yml puts the synced .venv/bin on PATH, which is the only way `poly lint .` finds it on a runner. Taskfile calls it through `uv run` for the same reason, so a contributor needs no system install.
`poly doctor` reports that an unanchored glob matches a directory of that name at ANY depth. `diagrams/**` and `epub/**` were therefore also hiding scripts/corpus_tools/diagrams and scripts/corpus_tools/epub — eight files this branch added that poly then silently stopped linting. It went from 70 files to 78 once anchored. The original config already anchored `/office/**` and `/MATH_PROVENANCE.md`, so the pattern was known; the rest were left unanchored and it did not matter until a package name collided with a corpus directory name. All 55 rules are anchored now, and a ~keep records why, since the next subpackage called `text` or `code` or `json` would hit the same thing. Verified the corpus is still excluded (52 directories, and code/hello.py and jupyter/mime.ipynb are both refused by name), and that an adversarial `poly fmt .` still changes nothing outside scripts/.
The same bug as the poly one, in the other config, and worse because nothing reported it. `extend-exclude = ["diagrams", "epub", ...]` excludes a directory of that name at ANY depth, so ruff was skipping scripts/corpus_tools/diagrams and scripts/corpus_tools/epub — eight first-party files, never linted, never format-checked, with four dead imports sitting in them. `poly doctor` surfaces the poly half; the ruff half was silent. Verified all five anchoring forms against a real tree: a bare `diagrams` excludes both the corpus directory and the source package; `/diagrams` is ignored by ruff's globset entirely, so it excludes neither; `./diagrams` and `diagrams/` exclude only the top-level one. Every entry now uses `./`, and poly.toml uses the leading slash it wants for the same list. Three tests hold it: both exclude lists must be anchored, and ruff must actually lint the two subpackages whose names collide with corpus directories. The sync test normalises the two anchoring styles rather than comparing raw strings, since neither tool accepts the other's form. poly went 70 -> 78 files linted; ruff found and fixed the four dead imports. Corpus protection is unchanged: 52 directories still excluded, and code/hello.py and jupyter/mime.ipynb are still refused by name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11, and adds the private-corpus support it was blocking.
Private corpus support
fetch_corpus.pygains--manifest,--rootand--auth;publish_corpus.pygains--manifest,--rootand--patterns. Every one defaults to today's value, so the public commands are byte-for-byte unchanged.Authentication is opt-in and anonymous stays the default — that is not a convenience, it is what keeps the credential-free CI proof meaningful. It uses Application Default Credentials via
gcloud, so a local login and workload-identity federation in CI both work with no configuration, and it re-acquires as the token ages because a large fetch on a slow runner can outlive one.Three guards, because publishing is irreversible and the public bucket cannot un-serve an object:
--rootmust not sit above where the explicitly-named paths put the corpus, unless--allow-external-rootsays you mean it;The second one matters more than it looks: corpus patterns use gitignore semantics, so a bare
*.zipmatches a basename at any depth and a root one level too high sweeps in whatever sits beside the corpus.Issue #11
download()existed five times,sha256_ofthree, and three of five call sites retried with no delay at all — the same request into the same transient failure, three times within milliseconds. All of it now sits behindscripts/corpus_tools/, and the ten entry points are 30–180 lines of docstring, parser andmain().Two transports stay, deliberately. curl is the path consumers take — the CI action fetches with curl — so verifying with urllib would leave CI green while asserting something about a TLS stack no consumer uses. The five unexplained timeouts became three with their derivation written beside them;
300sturned out to belong to exactly one URL.No consumer repository needs a change. All ten entry-point paths are frozen, the flag surface is purely additive, and
corpus.lock.jsonis byte-identical — so the cache keyxberg-io/actions/fetch-test-documentscomputes is unchanged. Submodule-bump reviewers can stop looking.Behaviour changes, declared
fetch_corpusandverify_corpus --samplegain retries (the CI flake surface).build_diagram_pdfs's hashing goes whole-file → chunked. Same digest, lower peak memory; the corpus holds a 62 MiB object.fetch_regression's error prefix gains the exception type — it previously printed a bareRuntimeErroras the word "error" and nothing else.Corpus protection
ruff formatreads Jupyter notebooks natively: unconfigured, it would rewrite four fixtures underjupyter/math/and fail to parse three more, andruff check --fixwould delete the two deliberately unused imports fromcode/hello.py.force-exclude = trueis the line that matters — without it any explicitly-named path bypasses the exclude list, which is what an editor-on-save does.test_lint_scope.pyasserts that behaviourally against the real binary rather than trusting the config, and cross-checks thatpoly.tomlandpyproject.tomlstill describe the same corpus. Verified by runningruff check --fix . && ruff format . && poly fmt .at the repo root: zero corpus files modified.Adopting
uvalso opened a publish hazard worth naming —corpus_paths()walks the tree ignoring git, and.venv/was neither pruned nor gitignored, so venv assets were reachable by a publish to the public bucket. Closed three ways, with tests that fail if any one is removed.Tests: 98 → 191
Six modules had none. Both runners still agree, and the stdlib one is kept working on purpose — it is the same "works on a bare checkout" property
verify_corpusdefends.Everything network-free, through injected seams rather than patched globals: the curl transport takes a
runner,get()takes asleep,AdcCredentialtakes aclock.Four reviews of this branch found real defects, all fixed here — including one I introduced: rewriting the SVG stripper around argparse silently dropped its "nothing to strip" guard, which would have let the regeneration loop write geometry fixtures identical to their parents. Two tests provably could not fail and are rewritten.
test_entry_points.pyexists because moving the EPUB tools brokefetch_epub_edge_cases.pyat import while ruff and all 186 tests stayed green.Also
.ai-rulez/with five critical rules, four context files and two local subagents — this was the only repo in the polyrepo with Python and no AI governance, soCONTRIBUTING.md's instruction to generate it produced nothing. Plus the poly gatepoly.tomlhas been describing with nothing enforcing it, a Taskfile (task setuphad nothing to run), and three stale doc references corrected.