Repository navigation
Conversation
Extend the reference worker's contract with the two remaining Valen task types alongside choice: noul (fixed or custom true/false wording, answered as P(true) plus the probability pair and confidence) and score (2-10 ordered levels, per-level branches decoded as a cross-level softmax with the expected level, legend and concentration confidence, formulas copied from the pinned reference). Mixed-type requests share one state encoding; usage keeps the logical/compute split. Choice behavior is unchanged (the image/text parity blocks match the merged records byte for byte); noul and score parity against the pinned reference passes at 1e-6 on RTX 5060. 47 CPU tests; compare_reference gains --case noul|score|all with new example requests.
The cardinality check's message read like the satisfied condition; name the requirement so the 500-side error is actionable. (cherry picked from commit 3afa240)
The noul/score blocks were originally recorded from a working tree, and the rebase onto current upstream main (the reference slice landed as InferMatrixLab#115) orphaned the earlier verification SHAs. All four parity blocks and the frontend byte-identity were re-verified verbatim at 3afa240; the records now reference that tree. (cherry picked from commit 28072dc)
… the PR Align the noul/score documentation with the d3 recipe convention: the recipe records the procedure, commands and fixtures, while measured run output — the observed responses, the verbatim noul/score parity blocks and their provenance SHAs — lives in the pull request description. The model README drops its duplicate request examples and states the compiler record rules and answer fields in compact bullets. Only content this branch added or rewrote changes; the InferMatrixLab#115 slice's records stay as they were.
Address the review findings on compare_reference: dispatch the per-type comparison on the type requested in the wire body (rejecting answers that drift to another shape instead of silently skipping the noul/score checks), and include the confidence delta in the pass criteria with its own tolerance verdict line.
The four compare_reference blocks were re-run verbatim from the committed tool at 507c448; every block now carries the confidence tolerance verdict and the criteria paragraph states the requested-type and confidence gates.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
The LoRA merge bf16(fp32(W) + 2*(B@A)) is verified bit-equal against the pinned peft 0.18.1 Linear.merge on synthetic tensors; the FP32->BF16 downcast mirrors the reference's own from_pretrained cast. Checkpoint keys decompose into LoRA pairs, vision-top blocks 20-23, merger and head with strict rejection of anything outside the pinned vision_top layout.
run_export writes model.safetensors (merged backbone in the base's own key layout, BF16 only), the fp32 head.safetensors, the copied base config and the valen_export.json contract through an incremental safetensors writer, tensor by tensor in sorted order, so host memory stays far below the 4.5 GiB artifact set. validate_export re-reads headers and digests and rejects tampered bytes, unlisted tensors, downcast-list drift and head-config drift. The tiny synthetic fixture proves determinism and path independence.
export() hashes checkpoint.pt and the base shard, checks them against training_metadata.json, valen_manifest.json and the checkpoint's own base_manifest (both record per-shard sha mappings), refuses any stage but the pinned vision_top layout, and records the revisions, repo and tool versions in the manifest sources. main() runs it as the one-command recipe. The env-gated regression (VALEN_CKPT/VALEN_BASE) exports the real artifacts twice and requires byte-identical output plus the pinned inventory; run on this host it passed in 3m33s with all four files identical across runs.
Add recipe/valen/verify_export_parity.py (spec 02, E2): it loads the original pinned stack (ValenExecutor: base + unmerged FP32 LoRA + checkpoint) and the exported stack (from_pretrained(weights/valen-export) backbone + DecisionHead rebuilt from head.safetensors, run through the pinned Valen class) serially, one subprocess each so the ~6.6 GiB VRAM footprint never overlaps, answers the recipe's existing text and image choice fixtures through the pinned Compiler, and gates on exact answer agreement plus probability deltas <= 1e-6. It also measures the merge drift of all 186 merged keys on the real tensors (research-001 method: unmerged PEFT semantics vs the exported BF16 GEMM, 256 bf16 probe rows, seed 0, CPU single thread), cross-checks every exported tensor bit-exactly against E1's merge math, fills the manifest's parity slot with the verdict and re-validates the artifact set. tests/valen/test_export_parity.py covers the pure contract CPU-only (bf16 ulp ruler, per-module drift against peft 0.18.1's Linear.forward, aggregation, case verdicts, parity record + slot filling) and wires the env-gated full-artifact regression (VALEN_CKPT/VALEN_BASE/VALEN_SRC + CUDA), asserting the recorded verdict's internal consistency rather than a desired outcome.
Add recipe/valen/export.md (usage, artifact inventory, manifest schema, merge decision, downcasts, head, consumption, pins/licenses, parity pointer) and recipe/valen/test_export_reference.py, which chains the tests/valen/test_export.py battery and the env-gated real-artifact determinism gate and validates an exported set through the tool's own validate_export. Link the export from recipe/README.md, src/models/valen/README.md and one line in reference.md.
Append the spec-02 export-parity block to recipe/valen/reference.md: the two serially loaded stacks (original ValenExecutor build vs the exported artifact set's documented consumption path), the verbatim gate output on the RTX 5060 host (text delta 7.06e-04, image delta 1.73e-02, exact answer agreement on both fixtures), the 186-module merge-drift aggregate (max one BF16 ulp per adapted module, ~53% bit-exact, worst relative drift 7.6e-3), the control checks (same-stack bit-identical re-runs, original stack matching the committed serving records, exported tensors bit-checked against the merge math), and the honest verdict: the declared 1e-6 tolerance fails, so the spec-02 fallback (separate FP32 LoRA export, unmerged serving) is triggered; the merged set keeps exact discrete answers and parity.status = failed in the manifest.
…iles The manifest cross-check read every artifact whole (read_bytes), pulling the ~4.5 GiB model.safetensors into host memory and contradicting the documented headers-and-digests-only contract. Sizes now come from stat() and digests from the module's streaming sha256_file, and the per-tensor safetensors header check reuses one handle per file instead of reopening per tensor.
A missing training_metadata.json silently skipped the checkpoint digest check right before a weights_only=False unpickle, and the base weights pins were conditional on both sides declaring a digest. Following the worker precedent (engine.py pins unconditionally), a missing training_metadata.json or an undeclared base digest is now an error; the doc's pin paragraph says so.
verify_sources_match_manifest checked the checkpoint sha and base revision strings but not the base weights digests the manifest records, although 392 tensors pass through the export byte by byte. The gate now re-hashes the base shard against sources.base.weights_sha256; tampered records or swapped shard bytes are rejected before any model loads.
- validate_export counts every directory entry, so a stray subdirectory no longer bypasses the exactly-four-files audit; - run_export cleans up its partial output on failure (removing a directory it created), so a failed run never blocks the retry; - classify_checkpoint_weights rejects vision blocks outside 20-23, not only sub-20 blocks, matching the pinned vision_top inventory; - drop the unused BACKBONE_PREFIX/VISUAL_PREFIX constants and the unused cuda mem_get_info free byte.
The fallback paragraph claimed the gate passes and the fallback is deliberately unimplemented, contradicting reference.md's recorded FAILED verdict (max probability delta 1.7e-2). It now states the failure and leaves the fallback decision to the spec owner (local ticket E4). The suite count is updated to the measured 105 passed / 2 artifact-gated skips.
The recipe regression script introspects tests/valen/test_export.py and injects fixtures by parameter name, but only knew tmp_path; the new streaming/cleanup tests take monkeypatch and failed under the script while passing under pytest. The battery now hands each test a fresh pytest MonkeyPatch (undone after the test) for that parameter.
9f55f55 to
cfce7ba
Compare
MrDongsls
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f55f55088
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| payload = torch.load( | ||
| checkpoint_path, map_location="cpu", weights_only=False, mmap=True |
There was a problem hiding this comment.
When --checkpoint points to a corrupted or attacker-supplied directory, its sibling training_metadata.json can simply declare the corrupted file's hash, so the preceding check is not a trust anchor. This call then uses weights_only=False, allowing pickle reducers to execute before any inventory validation; unlike ValenExecutor, the exporter never compares the file to the fixed CHECKPOINT_SHA256 in src/models/valen/engine.py. Verify the hard-coded pinned digest before this load (or avoid unrestricted pickle loading).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 5bad3c1: the exporter now enforces the worker's pinned CHECKPOINT_SHA256 (src/models/valen/engine.py) before torch.load, so training_metadata.json remains a consistency record only. Regression test test_export_rejects_checkpoint_below_the_worker_pin proves the rejection fires before any unpickling.
| passed = decision_exact and legend_exact and prob_ok and tokens_exact | ||
| if scalar_answers: | ||
| passed = passed and max_answer_delta <= tolerance |
There was a problem hiding this comment.
For the newly added noul/score cases, a regression in the worker's confidence formula can still make this harness exit successfully: max_conf_delta is computed and printed, but it is never included in passed. For example, matching probabilities, score, legend, and tokens with worker confidence 0.99 versus reference confidence 0.4 reports a 5.9e-01 confidence delta and still returns True, so the advertised reference comparison would not catch an incorrect response field.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed by 507c448 (the same fix as #146), which this branch includes since the rebase — the reviewed 9f55f55 was the pre-rebase tree: max_conf_delta is gated by its own confidence tolerance verdict line and included in passed.
| from frontend.valen import _verify_source_revision | ||
|
|
||
| source = Path(args.valen_source).resolve() | ||
| _verify_source_revision(source) |
There was a problem hiding this comment.
When the pinned Valen checkout has local modifications, this check still passes because _verify_source_revision in src/frontend/valen.py only runs git rev-parse HEAD. Both the original and exported subprocesses then import the same modified compiler/model code, so a shared local change (for example, a changed Valen.forward) can make the two stacks agree and fill a passing parity record even though neither used the claimed pinned source; require a clean tracked working tree (and account for relevant untracked files) before producing evidence.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 5bad3c1 and extended in b6490ae: both evidence harnesses (verify_export_parity.py and compare_reference.py) now reject a dirty pinned checkout — tracked modifications and untracked .py files — before any record is produced, and reference.md's pre-flight checks match. Regression tests cover both rejection paths.
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
Independent review — PR #147 "valen: export the trained stack to an unmerged safetensors artifact set"
Reviewer: independent read-only review (system1-omni-review skill). No repo modifications, nothing posted to GitHub. Review body drafted 2026-10-18.
Provenance
- Repository: ThinkFlowLab/system1-omni (ID 1386454853).
- Base:
main, livee66ff34996760a84ad56690f0ae4cf434e669e65(moved past the PR's recorded basef2d3d3289…; the PR's stated upstream base037c44b9…is the merge-base — main moved by the unrelated #111/#116 merges after branching). PR GitHub base field:f2d3d32897c1fe190fecf853a82a9456d3791462. - Head:
369202d60da14dd061936d03963671f78531a9da(spec/valen-export), re-fetched live before reporting; GitHub headRefOid matches. - Skill load record:
.agents/skills/system1-omni-review/SKILL.mdread from the local working tree, SHA-25658be3bc60806c4c5af7038a0c6a80fadb94f3360d714556a3e965c8903dde108(matches the pinned value). References actually read:docs/architecture.md(no export-specific clauses; export is recipe-level per the jemm/open_jev precedent),.agents/skills/system1-omni-review/references/repository-contracts.md,CONTRIBUTING.mdat the head,.agents/skills/self-review/SKILL.mdat the head (incl. Committed artifact hygiene), the Valen recipe and docs at the head. - Inspected scope: full
git diff origin/main...origin/pr/147(24 files, +4924/−144), deep read of the export-side files; the tree was extracted to/tmp/pr-review-2026-10-15/head(working tree untouched).
PR structure and stacking
- The branch stacks on open PR #146 (
model/valen-noul-score, head75efa400…): commits 1–6 of this branch are #146's line. This is disclosed first thing in the PR body ("merge #146 first … drops the noul/score commits automatically once #146 merges"). - Full diff vs main: 24 files, +4924/−144. Export-only delta (
origin/pr/146...origin/pr/147): 12 files, +3672/−16 —export_checkpoint.py(1063),verify_export_parity.py(567),export.md(212),tests/valen/test_export.py(1099),tests/valen/test_export_parity.py(389),recipe/valen/test_export_reference.py(158), plus reference.md/README/compare_reference/mkdocs/recipe-README touches. - Authored vs total: all 25 commits authored by the PR author (Mr.Dong / MrDongsls); authored == total. Above the 3,000-line threshold, CONTRIBUTING's self-review requirements apply: full self-review checklist checked by the contributor, split rationale present (stacked PR, serving split from export), validation across affected areas present. The body has no explicit component map/review order — minor readiness note only, since the layout is recipe/tests/docs and the Purpose section enumerates the artifacts.
- Merge order matters: until #146 merges, this PR's GitHub diff shows both lines. The export review below is based on the export-only delta plus the head tree.
Verdict
No blocking findings. The export side is unusually carefully engineered: pins enforced before the pickle is loaded, every artifact byte digest-pinned in the manifest, fail-closed validation of partial/tamper/stale states, deterministic streaming writer, and a parity gate whose executable form matches its declared semantics (tested bit-equal against installed peft). Findings below are low/informational.
Findings (by severity)
- Low — atomicity is exception-driven cleanup, not staging+rename.
run_export(recipe/valen/export_checkpoint.py:330-384) writes in place into a fresh-or-empty--outand removes partial artifacts on any raised exception (tested:test_run_export_failure_leaves_no_partial_output,test_run_export_failure_cleans_preexisting_empty_output). A hard-killed process (SIGKILL/OOM/power loss) can leave partial files, and the not-empty guard then blocks retries until an operator removes the directory. Documented behavior; no corruption risk becausevalidate_exportrejects any non-exact directory. Temp-dir + atomic rename would close the gap. - Low —
compare_casezip truncation nit (recipe/valen/verify_export_parity.py:95-98): the original/exported probability lists are zipped without an equal-length check; a malformed stage payload with a different list length would silently truncate the max-delta scan. Candidate order is checked and both sides are produced by the same tool, so this is a robustness nit in the evidence harness, not a gate criterion. - Low —
run_export(base_config_path=None)writes a set its ownvalidate_exportrejects (allowed set requiresconfig.json,export_checkpoint.py:690-709). Not reachable viaexport()/CLI/tests (all pass the config); API-surface consistency only. - Info — the manifest does not record the exporting code's revision.
sourcesrecords checkpoint SHA-256 + revision, per-shard base digests + revision + repo, and torch/safetensors/peft versions; the format stringvalen-checkpoint-export/2is the contract anchor and the PR pins the code — but a repo commit field insourceswould make the manifest self-contained for future audits. Note the asymmetry: the parity harness pins the Valen source revision and demands a clean checkout with no stray.py(_verify_clean_checkout), which is stronger than the #140 fixture-pinning precedent. - Info — the artifact set deliberately excludes tokenizer/processor files ("Five files, nothing else"). "Nothing else is needed to reconstruct the model" is accurate; reconstructing the serving stack still needs the pinned base for the processor/tokenizer. A one-line note in export.md could preempt confusion. Optional.
- Info — supported-models row not extended for the export. The Valen row (
docs/supported-models.md:18) is updated fornoul/scorebut does not mention the export recipe; defensible, since the row's native-worker column is still "Not supported" and the export is reachable viarecipe/README.mdand the mkdocs nav. Revisit when the native consumer lands.
Per-focus review
1. Export identity/pinning
- Pinned bytes:
manifest.filescarries size + SHA-256 for every non-manifest artifact:model.safetensors(~4.5 GiB),lora.safetensors,head.safetensors,config.json— four of four. The manifest itself (valen_export.json) cannot self-hash; itsparityslot is filled only by the tool, formatting-preserving, refusing silent overwrite (fill_parity_slot, tested idempotent + refuse-without-force). - Source pins, fail-closed, before any write or unpickle:
checkpoint.ptSHA-256 vs the worker's hard-codedCHECKPOINT_SHA256(src/models/valen/engine.py:22, value matches the PR body and the env-gated test assertion836622ef…f42f5a012);training_metadata.jsonmust exist and agree; base shard digest must be declared by bothvalen_manifest.jsonand the checkpoint's embeddedbase_manifestwith exact per-shard mapping, and revisions must agree;stagemust bevision_top. Missing pins raise; they are never skipped (tests cover each missing/mismatched pin individually). This is the review-bot P1 from an earlier round, fixed and regression-tested ("the exporter must enforce the worker's pinned digest before the pickle is ever loaded"). - Rejection of stale/partial exports:
validate_exportenforces exactly-five-files directory contents (extra files or directories fail), format match, streamed file digests + sizes, full two-way tensor inventory vs safetensors headers with shape/dtype, downcast list vs per-tensor flags, head-config consistency, and the adapter inventory (one A + one B per target at the declared rank, each mapping onto a listed backbone tensor).verify_export_parityadditionally re-hashes the checkpoint and base shard against the manifest'ssourcesbefore loading (a swapped base fails:test_verify_sources_rejects_swapped_base_shard_bytes). - #140 lesson (pin executables/config too): the parity harness pins the Valen source revision (
frontend.valen._verify_source_revision) and rejects dirty checkouts or untracked.pyfiles before evidence is produced. The export itself pinsconfig.jsonbytes. No tokenizer is part of the artifact set (finding 5).
2. Unmerged-ness semantics
- The set preserves original base weights + separate FP32 adapters + separate FP32 head:
model.safetensors= base verbatim (578 pass-through, of which 36 FP32linear_attn.A_log/norm.weightdowncast to BF16) with trained vision-top blocks 20–23 and merger substituted as plain BF16 replacements (full tensors, not merges);lora.safetensors= 372 FP32 adapter tensors verbatim under PEFT key names;head.safetensors= 2 FP32 matrices. Exporting a merged path was deliberately dropped (commit "export the language LoRA separately and drop the merged path"), with the merge-vs-unmerged math documented (export.md: "Merging … changes the computed outputs relative to the trained artifact's unmerged semantics, which is why the export does not do it"). - BF16 rounding locations are explicit and machine-checked:
LORA_APPLICATIONis recorded verbatim in the manifest (y = bf16(fp32(bf16(x @ W.t())) + scaling·((x.float() @ A.t()) @ B.t()))— base GEMM BF16, delta FP32, exactly one final rounding, peft 0.18.1 unmerged semantics with FP32 adapters) andunmerged_lora_outputis its executable form, tested bit-exact against installed peft plus a hand-computed rounding case and a zero-B reduction case. Downcasts are claimed lossless against the reference's own BF16 runtime and gated: any FP32 base tensor outside the two declared families is an error. Head math stays FP32 with formulas recorded. - Loader-side contract: the manifest's
consumptionblock + export.md's "Consumption" section specify the full load (from_pretrained backbone, peft re-wrap at manifest rank/alpha loadinglora.safetensorsby key with adapters kept FP32, head rebuild). The native consumer is future work and is disclosed (PR body: "the native worker is separate upcoming work"; export.md: "serving from it is the consumer's side"; src/models/valen/README.md: "Consuming that export does not change this worker"). The documented consumption path is exercised for real only in the author's GPU parity run (see unverified checks).
3. Determinism / reproducibility
- Sorted keys everywhere (classification, tensor plans, manifest JSON
sort_keys=True), streaming writer with fixed per-file ordering, single-threaded CPU math, no timestamps, no absolute paths — tested byte-identical across runs and across different input directory names (test_run_export_is_deterministic_and_path_independent, run locally). The real-artifact double-export determinism gate is env-gated (VALEN_CKPT/VALEN_BASE) — author-run, with per-file digests recorded in the PR body. - Source revisions pinned (checkpoint SHA-256 + HF revision when derivable, base per-shard digests + revision + repo, tool versions in
sources.environment). Exporting code pinned by the PR head + FORMAT version (finding 4 for the residual gap).
4. Docs
recipe/valen/export.mdis complete and matches the code: usage + expected console output, artifact table with exact counts (578/372/48/6/2 = 1006 manifest tensor records), manifest schema, LoRA application semantics, downcast rationale, head formulas, consumption, pins/licenses (Apache-2.0 per the #115 record, with a re-check instruction), parity-evidence placement, determinism + regression commands with recorded counts ("105 tests passed, 2 artifact-gated skips" — matches my run).reference.mdgains a full "Export parity" section with verbatim tool output (RTX 5060 Laptop, sm_120, bit-level deltas 0.0, recorded 2026-10-15) and the reproduce command; the honest "~3e-8 worker-vs-reference spread" between the parity run's original stack and the earlier serving records is disclosed. Anchors referenced by export.md exist.src/models/valen/README.mdadds a "Checkpoint export" section;recipe/README.md, mkdocs nav, README matrix and supported-models updated for noul/score. Template readiness: complete (see below).
5. Tests
- Weight-free coverage is strong and CPU-only: unmerged-vs-peft bit equality, hand-computed BF16 rounding, downcast RNE, key classification incl. adversarial cases (block 24 swap, unpaired adapters, unknown keys, head bias), tiny-fixture end-to-end export contract, path-independent byte determinism, failure cleanup, per-pin rejection tests,
validate_exporttamper matrix (flipped bytes, dropped tensor entries, downcast/head/lora drift, extra files/dirs), streaming-digest enforcement (monkeypatchedPath.read_bytesrefusal), parity verdict matrix, parity-slot overwrite policy, source-pin cross-checks, clean-checkout gate. - Fixture provenance: all synthetic, seeded, generated in-test; the two example request JSONs are maintained fixtures consumed by reference.md curl commands and
compare_reference.py— no committed run output anywhere in the 24 files (all text source/docs/fixtures, no binaries).
6. Artifact hygiene (all 24 files)
16 modified + 8 added files; all are source, tests, docs, or maintained request fixtures. No exported weights, no parity dumps, no logs, no site/ build output committed. Authored counts = total (all commits by the PR author).
Commands run (this host, macOS arm64, no GPU, no Rust)
All on the extracted head tree /tmp/pr-review-2026-10-15/head with a throwaway venv (Python 3.12.8; torch 2.14.1 CPU, safetensors 0.8.0, peft 0.21.2, pytest 9.1.1, pillow — not the author's pinned GPU matrix, torch 2.11.0+cu128 / peft 0.18.1):
git fetch origin main --force→ live maine66ff34996760a84ad56690f0ae4cf434e669e65;git fetch origin pull/147/head:refs/remotes/origin/pr/147 --force→ head369202d60da14dd061936d03963671f78531a9da(matches GitHub).shasum -a 256 .agents/skills/system1-omni-review/SKILL.md→58be3bc6…dde108(pin verified).PYTHONPATH=src python -m pytest tests/valen -q→ 105 passed, 2 skipped in 4.99s — matches the author's claim and export.md verbatim. Skips are exactly the artifact-gated gates (test_export.py:1054needs VALEN_CKPT/VALEN_BASE;test_export_parity.py:348needs VALEN_* + CUDA).- Export-only subset
pytest tests/valen/test_export.py tests/valen/test_export_parity.py -q→ 58 passed, 2 skipped. python recipe/valen/test_export_reference.py→export_reference tests: ok (2 skipped)(full 39-test battery in-process + skip lines for artifact-gated phases, exit 0).python recipe/valen/test_compare_reference.py→compare_reference tests: ok(incl. the new score-confidence-fallback case).python -m compileall -q recipe/valen tests/valen→ pass.python -m mkdocs build --strict→ exit 0 (done because the Docs workflow evidence was initially not visible to me; it turned out green at head — see below).gh pr checks 147/ check-runs API at head369202d→ rust: pass (2m0s), build: pass, benchmarks: pass, cuda-contract: pass, deploy: skipped, Docs: pass (run 2026-10-15T15:21:57Z at the head SHA). Note: no GPU/model validation runs in CI; the rust/docs results are consistent with a Python/docs-only change.
Template state (post-#138)
Complete and traceable: the four required human answers are present and specific; the PR-necessity decision answered ("Yes, without this every consumer converts the pickle on its own"); bug-fix evidence marked N/A (not a bug fix); Purpose/Test Plan/Test Result/Demo-evidence filled with pinned head SHA (369202d), environment, per-file manifest digests, and verbatim parity output; all four self-review checkboxes checked by the contributor; the stacking on #146 and the not-yet-implemented native consumer are both disclosed. NON-draft, so template completeness was required — it is met.
Material unverified checks
- GPU parity evidence is author-run, not independently reproduced (this host has no GPU, no pinned weights): bit-level 0.0 deltas on the text/image fixtures, RTX 5060 Laptop (sm_120), torch 2.11.0+cu128, peft 0.18.1. It is recorded verbatim in
reference.md(with reproduce command) and the manifestparityslot; the env-gated tests (test_real_artifact_export_is_deterministic,test_real_artifact_export_parity_gate) are the reproducible form and were skipped here. Per the skill: missing GPU/weights is an evidence gap on my side, not a PASS by me. - The documented consumption path executed against real weights (
from_pretrained+ peft re-wrap key-name mapping + head rebuild) ran only in the author's parity run; there is no weight-free test ofload_exported_stack(inherent — it needs transformers and a real model). The PEFT key-name round-trip (file keysbackbone.language_model.base_model.model.…vs wrapped-model parameter names through the top-levelValencontainer) reads consistent and is covered end-to-end by the author's run, but I could not execute it. - Real-artifact byte determinism — author-run (digests in the PR body); env-gated test skipped locally.
- transformers-internals claim in export.md ("no keep_in_fp32_modules for this architecture; upcasts with .float() at use") — not independently verified against the pinned transformers version here; it is corroborated indirectly by the bit-level parity result.
- Rust checks: not run locally (no Rust toolchain on this host); CI rust job green at head. My local Python versions differ from the author's pinned matrix (noted above); the peft bit-equality test passing on peft 0.21.2 is a small independent corroboration of the declared semantics, not a substitute for the pinned-env GPU run.
Bottom line
The export half is ready on the evidence presented, modulo the disclosed future consumer: pins enforced before load, every artifact byte pinned, fail-closed validation, deterministic output, semantics anchored and tested. Merge-order dependency on #146 is real (the GitHub diff collapses after #146 merges). The low findings are polish; none block on correctness.
Review follow-ups on the export recipe, kept minimal: - run_export now stages the artifact set in a sibling directory and moves it into place with one rename, so a hard-killed run can no longer strand partial files under --out (a stale staging directory is cleared by the next run with the same target). - run_export rejects base_config_path=None instead of writing a set its own validate_export refuses (the allowed set includes config.json). - compare_case fails a case whose probability lists do not cover every candidate instead of silently truncating the delta scan. - export.md notes the artifact set deliberately excludes tokenizer and processor files (reconstructing the serving stack reads them from the pinned base) and records the new suite count (108 passed, 2 skipped).
|
Thanks for the thorough review. All three Low findings are addressed in
On the Info items:
Regression coverage: three new tests ( |
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
Incremental re-check at new head d1236c661b: one new commit since my review at 369202d60d — "valen: stage the export atomically and gate compare_case on list lengths" (5 files, +95/−24); the 25 reviewed commits are patch-identical (git range-diff, named refs, cat-file verified).
All three Low findings from my review are fixed, verified in the diff and by execution:
- Atomicity — fixed:
run_exportnow stages the artifact set in a sibling.export-partialdirectory and moves it into place with a single rename, soout_dironly ever holds a complete export; a killed run leaves at most the staging directory, which the next run with the same target clears. - compare_case zip — fixed: a case whose probability lists don't cover every candidate now fails the case instead of silently truncating the delta scan.
- base_config_path=None — fixed:
run_exportnow raises instead of producing a set its ownvalidate_exportrejects.
Also carried: the tokenizer/processor exclusion got its documented one-liner in export.md, and the suite count was updated. Locally run in a fresh CPU-only venv at this head: 108 passed, 2 skipped (exactly the doc's updated count; the 2 skips are the artifact-gated determinism gates). mkdocs build --strict passes; fork CI (rust/build/cuda-contract/benchmarks) green at this exact head.
No actionable findings remain. APPROVED. Unverified checks (unchanged from my first review): the RTX 5060 bit-level parity and real-artifact double-export determinism remain author-run (no GPU/weights on this host — the 2 skips are those gates); the documented consumption path against real weights exists only in the author's run; Rust suite covered by exact-head CI. Provenance: review skill at main@71857e0 (SHA-256 58be3bc6…dde108); remote head re-pinned immediately before posting.
|
[P2] NaN probabilities are accepted as zero-error export parity. At verify_export_parity.py:109, Python's argmax also retains the first candidate in this example, so the discrete gate does not rescue the result. Please reject non-finite probabilities on both sides before subtraction/argmax, validate probability ranges/normalization, and add regressions for NaN in either side and matching infinities. Keep non-standard numbers out of serialized evidence as well. Personally run: 108 CPU tests passed, 2 artifact-gated tests skipped, plus the failing NaN probe above. Environment: Python 3.11.15, torch 2.14.1+cpu, PEFT 0.18.1, Transformers 5.19.0, safetensors 0.8.0. This differs from the author's GPU/reference environment and is only CPU-tooling evidence. Strict docs passed. No pinned Valen/Qwen2B artifacts or real dual-stack GPU export comparison were available locally; historical author parity is not re-certified. GitHub currently reports conflicts with main. Readiness: the effective source/test diff is over 3,000 changed lines; please also supply separate authored/total counts, split rationale and component review order under current CONTRIBUTING. This is preparation feedback, separate from the numerical defect above. Review provenance and scope
|
|
Fixed in |
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
Independent review — PR #147 "valen: export the trained stack to an unmerged safetensors artifact set" — incremental delta re-check
Review scope provenance. Local repo /Users/hsliu/thinkflowlab/system1-omni. Prior review: APPROVED, 2026-10-18 (pass-64) at head d1236c661bf90dcbdcda3928508e84b424486272 — all three Lows verified fixed, 108 passed + 2 skips verified locally. Live head at this re-check: 7243b6750062242f7e1b40be76837fe3168e94cd. origin/main re-pinned at a893f0cf (#131 merge); merge-base(origin/main, head) = f5ae5acf. GitHub diff vs merge-base: 24 files, +5108/−144 (recomputed exactly — matches the body's Large-change packaging line). Non-draft, OPEN, base main. Stacks on #146: this head contains #146's line (77945e9b5) as c94caf80's second parent — merge #146 first.
Own-series integrity (gate 1)
First-parent chain from the previously approved head (git log --first-parent d1236c661..7243b6750, all commits cat-file -e verified):
c94caf80— merge "Merge branch 'model/valen-noul-score' into spec/valen-export";c94caf80^1==d1236c661byte-identical (no rewrite of the approved series); second parent is #146's new head77945e9b5(which itself absorbs upstream/mainf5ae5acf— that is the "absorbed main" in the delta).7243b675— the only new authored commit: "valen: reject non-finite probabilities in the parity comparison" (4 files, +115/−2).
Everything else in the +18444/−1200 overall delta is main-side content entering through the #146 line. Nothing enters that is not main-side or own-side: git diff 77945e9b5 7243b6750 --stat = 12 files, +3856/−16 (the export delta plus the authored commit); git diff f5ae5acf 7243b6750 collapses to exactly the 24-file PR footprint.
Merge hygiene (gate 2)
git merge-tree --write-tree d1236c661 77945e9b5 reports one content conflict: mkdocs.yml (both sides added a nav line at the same spot). The resolution was read in full: the −3 relative to the conflicted auto-merge content is exactly the removal of the three conflict-marker lines — both content lines survive (final file, lines 69–72: Valen reference worker, Valen checkpoint export, Cua-S1 native screenshots, Cua-S1 vision CUDA Graph), and recipe/cua_s1/native_screenshot.md exists in the tree. Correct content union; no leftover markers anywhere (git grep); the merged nav is validated by the green mkdocs build --strict Docs run at this exact head (below).
Authored commit 7243b675 — full review
The commit hardens both parity gates against non-finite probabilities — the machinery my APPROVE relied on:
recipe/valen/verify_export_parity.pycompare_case: after the candidate-order and probability-length gates and before the zip/argmax work, any non-finite (math.isfinite) value on either side appends a named problem, setsexact = False, andcontinues. Verified fail-closed end-to-end: the verdict'spassed = exact and tokens_agree and tolerance_pass. Placement is correct (the length gate still runs first; the new gate cannot be bypassed by a shorter list).recipe/valen/compare_reference.pycompare_case: same gate after the answer-type and candidate-order gates, before the delta scan and the per-type (noul/score/choice) comparisons; setsdecision_exact = False, sopassed(which requiresdecision_exact and legend_exact and prob_ok and confidence_ok and tokens_exact[+ answer tolerance]) fails. Rejection is in the comparison gate, not the exporter — matching the commit message.- Exporter-side defense in depth:
fill_parity_slotand therun_stagepayload dump now serialize withallow_nan=False, so a non-finite value cannot even reach a stage file or a manifest parity slot — it fails loudly at write time. - Tests pin it: three new checks in
recipe/valen/test_compare_reference.py(worker NaN fails, reference NaN fails, matching infinities on both sides still fail — closing theinf − inf = NaNand equal-argmax holes) and two new tests intests/valen/test_export_parity.py(exported NaN fails; matching infinities fail), assertingpassed is False,exact_answer_agreement is False, and the problem string. The asserted verdict fields exist in the verdict dict as read at this head.
New finding
P3-1 — the finite gate is asymmetric: scalar answer and confidence deltas still swallow NaN via max().
Where: recipe/valen/compare_reference.py:259 and :274 (max_answer_delta = max(max_answer_delta, abs(answer["noul"|"score"] − ref))), :266/:285/:292 (max_conf_delta).
Trigger→consequence: CPython's max(a, b) keeps a unless b > a; abs(NaN) > a is False, so a NaN delta is silently dropped and the answer tolerance / confidence tolerance lines PASS — exactly the failure mode named in the commit's own comment ("a NaN difference keeps the previous finite maximum"), surviving one layer over in the same function on the scalar paths the commit did not cover. A worker response (Python json.loads accepts NaN/Infinity literals) with finite probabilities but a NaN score, noul or confidence field would pass those gates.
Why only Low: not reachable from the pinned worker — the scalar answers and confidence are derived by postprocess.py from the same probability vectors the new gate now proves finite; the reference-side confidence values were inspected finite in the prior round; and verify_export_parity.py has no scalar path at all (and is allow_nan=False at both writes). Same evidence-harness robustness class as the zip-length Low fixed in pass-64.
Suggested fix: extend the same math.isfinite sweep to answer["confidence"], answer["noul"], answer["score"] and any reference-carried confidence before the per-type branches (one small helper, plus one NaN-confidence regression per side).
Note (no action required beyond the fix above): recipe/valen/export.md:209 still says "108 tests passed, 2 artifact-gated skips at the time of writing" — the suite is now 110 + 2 (the body's Test Result correctly says 110). The qualifier keeps it honest, but the author's own practice (the count was refreshed at d1236c661) is to keep it in sync; refresh alongside the P3 fix.
Prior findings — status at the new head (re-verified, not copied)
All three pass-64 Lows re-verified intact in the new tree; the authored commit touches none of them:
- Atomic staging — intact:
run_exportstill stages in the sibling.export-partialdirectory and installs with a singleos.replace(staging, out_dir)(export_checkpoint.py:366–388), withexcept BaseExceptionstaging cleanup and stale-staging removal on retry. compare_casezip length gates — intact: explicit length-vs-keys check inverify_export_parity.py(runs before the new finite gate);compare_reference.py's candidate-order check doubles as its length gate before the zip.base_config_path=None— intact: raises "base_config_path is required…" (export_checkpoint.py:359–361).
The pass-63 informational items (manifest does not record the exporting code revision; tokenizer/processor exclusion documented; supported-models row not extended) are unchanged and remain notes.
Template readiness and body pinning (post-#138 gate)
All four required human answers, the PR-necessity answer, bug-fix N/A, and all four self-review checkboxes present. The body was re-pinned to the new stack: head 7243b67, based on 77945e9, "which merges current main f5ae5ac"; the Large-change packaging section cites git diff --numstat f5ae5acf..7243b675 and its headline (24 files, +5108/−144) is exactly correct; component review order updated for "non-finite rejection". Test Result claims pytest tests/valen: 110 passed, 2 artifact-gated skips at this head — author-reported; statically consistent (pass-64's verified 108 + the 2 new tests), not independently executed this pass (see below).
Commands run and results (this host: macOS arm64, Python 3.12.8, no GPU, no Rust)
- Worktrees at both heads (removed after):
PYTHONPATH=. python -m pytest tests/benchmarks -q→ 65 passed at7243b6750(and at77945e9b5). PYTHONPATH=src python -m pytest tests/valen -q --ignore=tests/valen/test_export.py --ignore=tests/valen/test_export_parity.py→ 47 passed at this head.python recipe/valen/test_compare_reference.py→compare_reference tests: ok— includes the three new non-finite checks, so the new gate logic is exercised on this host.- Series/merge forensics: parent-hash equality,
merge-tree+ resolution diffs, collapse-to-footprint stats,case_side/verdict-dict cross-checks (all above). - Blocked, environment-side: the export battery (
tests/valen/test_export.py,test_export_parity.py,recipe/valen/test_export_reference.py) could not execute this pass — every torch install on this host is broken (libtorch_cpu.dylibmissing from the installed wheels) and PyPI is unreachable from the sandbox, so no replacement venv could be built. This is a gap on my side; those tests were executed and verified in pass-64 and the only changes since are the two reviewed additions. - PR bodies fetched via API; check-runs + workflow-runs queried for both head SHAs in both ThinkFlowLab and InferMatrixLab.
CI observed at this exact head
gh pr checks 147 plus the Actions/check-runs APIs at 7243b6750062… in both orgs: rust pass (2m12s), build pass, benchmarks pass, cuda-contract pass, Docs pass (mkdocs --strict), deploy skipped (main-only). All five non-skipped checks success at the exact head SHA in ThinkFlowLab and InferMatrixLab. CI does not execute tests/valen.
Material unverified checks (carried from the prior approval)
- RTX 5060 bit-level parity (0.0 deltas, both fixtures) and real-artifact double-export determinism at this head — author-reported (body says both were re-run at
7243b67); no GPU/weights here; the env-gated tests remain the reproducible form. - The full-suite 110-count and the export battery at this head — author-reported; statically consistent with my pass-64 execution plus the two reviewed tests; not executed this pass (torch environment gap above).
- The documented consumption path against real weights — author-run only (unchanged, disclosed).
Verdict
COMMENTED — downgraded from APPROVE: one new P3 (NaN-swallowing residue on the scalar answer/confidence delta paths of compare_reference.py, the same failure class this PR's new commit fixes for probabilities). Everything else is clean: series identity byte-verified, merge resolution correct, prior Lows intact, template re-pinned and complete, CI green at the exact head in both repos. Fixing the P3 (a small same-pattern extension of the new gate, plus the optional export.md count refresh) restores approval-eligibility. Per the standing rules this report is not posted anywhere and does not modify draft state, labels, or checkboxes.
|
P3 addressed in
Re-run from the committed tree at |
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
Independent review — PR #147 "valen: export the trained stack to an unmerged safetensors artifact set" — incremental delta re-check
Review scope provenance. Local repo /Users/hsliu/thinkflowlab/system1-omni. Prior review: COMMENTED (pass-93, 2026-10-24) at head 7243b6750062242f7e1b40be76837fe3168e94cd — one P3 (NaN-swallowing scalar answer/confidence deltas in compare_reference.py) plus a stale export.md test count. Live head at this re-check: 70478332ae6fa0fb87a4b8b589ed433a43bb37fc (= gh pr view 147 --json headRefOid). Non-draft, OPEN, title unchanged, base main. Stacks on #146 (head contains #146's line via c94caf80, unchanged from prior passes) — merge #146 first.
Own-series integrity (gate 1)
git log --first-parent --oneline 7243b6750..70478332a contains exactly one commit: 70478332 "valen: reject non-finite scalar answers and confidences in the parity comparison" (3 files, +104/−1). Parent check: 70478332^ == 7243b6750062242f7e1b40be76837fe3168e94cd byte-identical (exact full SHA) — the previously reviewed series, including the pass-93 probability-finite commit 7243b675, was not rebased or rewritten; no range-diff needed. All seven relevant SHAs (prior head, new head, prior heads of #151/#152, origin/main a893f0cf) verified with git cat-file -e. Commit read in full (git show).
The authored commit 70478332 — closes pass-93 P3-1 in full
Coverage of every path the pass-93 P3 flagged. New helper scalar_answer_fields (recipe/valen/compare_reference.py:191-213) collects, and the new gate in compare_case (:273-286) sweeps for math.isfinite before any per-type branch runs:
- worker
answer["confidence"]— always, so themax_conf_deltafold at all three prior sites (:304noul,:323score,:329choice) is covered regardless of answer type; - worker
answer["noul"]/answer["score"]for the respective expected types — covers themax_answer_deltafolds (:297,:312); - reference-carried
noul/score/confidencewhen present inref_answer— covers the reference sides of the same folds.
Every remaining source of a compared scalar is derived from the probability vectors, which the pass-93 commit 7243b675 already gates (the choice-shaped noul decode reads ref_answer["probabilities"], the fallback confidences are formulas over gated ref_probs), so no compare path can absorb a non-finite value anymore.
Fail-closed semantics — same as the probability fix. Placement is after the answer-type, candidate-order and probability gates and before the delta scan; a hit appends a named problem, sets decision_exact = False, and continues. Verified end-to-end that passed = decision_exact and legend_exact and prob_ok and confidence_ok and tokens_exact (+ answer tolerance when scalar_answers) at :378-382 — the verdict fails closed. No regression to the probability gates: the diff leaves them untouched, the battery still exercises worker-NaN / reference-NaN / matching-infinity probability cases, and the weight-free valen subset and benchmark counts are unchanged from pass-93 (below).
Tests pin it. Three new checks in recipe/valen/test_compare_reference.py (worker NaN confidence in choice shape, worker NaN noul, reference NaN confidence in score shape), each asserting not ok plus the new problem string, the first also "decision agreement: FAIL". All three observed passing locally; the standalone battery is 35/35 ok at this head.
The export.md staleness note — also addressed. recipe/valen/export.md:209 now says "110 tests passed, 2 artifact-gated skips" — accurate: this commit adds no pytest-collected test (the new checks live in the standalone recipe/valen/test_compare_reference.py script), and a static count of tests/valen gives exactly 112 test defs = 110 + 2, matching the body's Test Result.
Prior findings — status at the new head
- Pass-93 P3-1 (scalar/confidence NaN swallowed by
max()) — RESOLVED, verified in full above. - Pass-93 note (
export.md"108 tests") — RESOLVED (now 110, and now correct). - All three pass-64 Lows re-spot-checked intact: atomic staging + single
os.replace(export_checkpoint.py), zip/candidate-order length gates (unchanged, run before the new gate),base_config_path=Nonerefusal. Pass-63 informational items unchanged (manifest does not record the exporting code revision; tokenizer/processor exclusion documented; supported-models row not extended).
Body / evidence check at the new head
Template complete: the four required human contributor answers, the PR-necessity answer, the bug-fix N/A, and all four self-review checkboxes are present. The body is re-pinned to the new head: Version/Commit says "head 70478332 on spec/valen-export … CPU checks were re-run from this committed tree, and the determinism gate … re-run against the pinned artifacts at this head"; Test Result adds "compare_reference.py --case all on the GPU host: 4/4 cases … re-run at this head after the non-finite scalar gate" (author-reported); the component review order now cites "non-finite scalar rejection". The Large-change packaging headline was re-measured and recomputes exactly: git diff --numstat f5ae5acf..70478332a = 24 files, +5210/−143, matching the body verbatim; the authored/excluded split (+4702/−102 across 15 vs +508/−41 across 9) is arithmetically consistent with it.
Commands run and results (this host: macOS arm64, miniconda Python 3.12.8, no GPU, no Rust)
- Worktree at
70478332a(removed after):PYTHONPATH=src python -m pytest tests/valen -q --ignore=tests/valen/test_export.py --ignore=tests/valen/test_export_parity.py→ 47 passed (unchanged from pass-93 — no regression). python recipe/valen/test_compare_reference.py→ 35/35 ok, including the three new non-finite scalar checks and the pass-93 probability checks (worker NaN, reference NaN, matching infinities).PYTHONPATH=. python -m pytest tests/benchmarks -q→ 65 passed (unchanged).- Static count of
tests/valen(112 defs) to confirm the refreshedexport.md/body 110+2 claim — the torch suite itself remains blocked environment-side (every torch install on this host is broken:libtorch_cpu.dylibmissing; unchanged since pass-93), so the full-suite run stays author-reported. - Series forensics: parent-hash equality, first-parent log,
cat-file -eon all SHAs, numstat recompute (above).
CI observed at the exact head
gh pr checks 147 (ThinkFlowLab): rust pass (2m13s), build pass, benchmarks pass, cuda-contract pass; deploy skipped (main-only). Actions API queried at the full SHA 70478332ae6fa… in both ThinkFlowLab and InferMatrixLab: CI and Docs both completed / success with head_sha matching the PR head exactly. CI does not execute tests/valen.
Notes (informational, no action required)
- The new scalar tests pin NaN on both sides but not ±∞ explicitly; the gate uses
math.isfinite, so infinities take the identical path, and the probability-side matching-infinity test (test_compare_reference.py:302-309) still pins that pattern one layer down. Reference-side NaNnoul/scoreare not directly exercised (only reference NaN confidence), but they flow through the same sweep loop as the tested field. scalar_answer_fieldsindexesanswer["confidence"]/answer["noul"]/answer["score"]unconditionally — the same shape assumption the per-type branches below it have always made; no new failure mode.
Material unverified checks (carried; unchanged from the prior approval basis)
- RTX 5060 bit-level parity (0.0 deltas) and double-export determinism at this head — author-reported (body says both re-ran at
70478332); no GPU/weights here; the env-gated tests remain the reproducible form. - The full-suite 110-count execution at this head — author-reported; statically consistent (112 defs; pass-64 executed 108+2; the two export-file tests added since were reviewed) — not executed this pass (torch environment gap).
- The documented consumption path against real weights — author-run only (disclosed).
Verdict
APPROVED — restored from pass-93's COMMENTED. The single open P3 is fixed in full: every flagged scalar/confidence delta path is now swept with the same fail-closed gate semantics as the probability fix, pinned by three new passing checks, and the export.md count is refreshed and accurate. Series identity byte-verified, no new findings, template complete and re-pinned, CI green at the exact head in both orgs, local weight-free suites green and unchanged. The remaining author-reported items (GPU parity, determinism, full-suite execution) are the same ones the pass-64 APPROVE accepted as disclosed.
|
Follow-up at compare_reference.py:439–443 rejects untracked files only when their names end in CPU-only reproduction from this PR checkout; the temporary module only sets a harmless marker: import pathlib, py_compile, subprocess, sys, tempfile
sys.path.insert(0, "recipe/valen")
import compare_reference as c
with tempfile.TemporaryDirectory() as directory:
root = pathlib.Path(directory)
subprocess.run(["git", "init", "-q", str(root)], check=True)
(root / "README.md").write_text("fixture\n")
subprocess.run(["git", "-C", str(root), "add", "README.md"], check=True)
subprocess.run([
"git", "-C", str(root), "-c", "user.name=Fixture",
"-c", "user.email=fixture@example.invalid",
"commit", "-qm", "fixture",
], check=True)
source = root / "peft.py"
source.write_text("REVIEW_MARKER = 'unverified module imported'\n")
py_compile.compile(str(source), cfile=str(root / "peft.pyc"), doraise=True)
source.unlink()
print(subprocess.check_output(
["git", "-C", str(root), "status", "--porcelain"], text=True))
c._verify_clean_checkout(root) # observed: accepts
print(subprocess.check_output([
sys.executable, "-c",
"import sys; sys.path.insert(0, sys.argv[1]); "
"import peft; print(peft.REVIEW_MARKER)",
str(root),
], text=True))Observed: Git reports Next step: close the importable-file gap (source, sourceless and extension modules/packages, including relevant ignored entries), or run the reference from a clean snapshot materialized from the pinned commit. Add the sourceless regression under root A separate readiness cleanup: the newly added Personally executed on the candidate merge with current main: 110 Valen tests passed, 2 artifact-dependent tests skipped; 35 comparison checks passed; export regression script passed with real-artifact phases skipped; compileall, diff check and strict MkDocs passed. PyTorch 2.11.0+cpu / PEFT 0.18.1 / Python 3.12.13. GPU parity and real-artifact determinism were not rerun. Authored code/test diff +4702/−102; total +5210/−143 across 24 files; diff merge-base Review provenance: |
…s in the parity comparison Forward-port the finite-value guards from spec/valen-export (7243b67, 7047833) into this prerequisite branch. max() keeps its first argument against a NaN difference and a NaN-free argmax list silently keeps its first candidate, so a non-finite probability, scalar answer or confidence on either the worker or the reference side fails the case before any delta, argmax or tolerance verdict. Regressions cover worker NaN probabilities, reference NaN probabilities, matching infinities, worker NaN confidence, worker NaN noul answers and reference NaN confidence. The export-side harness (verify_export_parity.py) and its tests stay on spec/valen-export, as does the score-confidence formula fallback.
# Conflicts: # recipe/valen/compare_reference.py # recipe/valen/test_compare_reference.py
…-source gate _verify_clean_checkout only rejected untracked .py files: a standalone .pyc (or a .so extension module) imports by name exactly like its .py source, and the checked-out directory is prepended to sys.path, so unverified bytecode could shadow a dependency while the run still claimed clean pinned reference code. The sweep now covers every importable suffix from importlib.machinery (source, bytecode, extension) over untracked *and* ignored entries (a checkout-local ignore rule hides from plain status), while __pycache__ caches stay allowed since they are not importable on their own. Both evidence entrypoints share the gate: verify_export_parity.stage_environment calls reference._verify_clean_checkout. Regressions cover the reviewer's sourceless-module repro, an ignored .pyc, an interpreter-tagged extension module, the __pycache__ allowance and the shared-gate pin. reference.md's pre-flight check mirrors the widened rule.
…alen The export regression runner (contract-battery chaining plus the env-gated artifact-set validation) lived entirely in recipe/valen/test_export_reference.py, putting test implementation outside the root tests/ tree. The implementation now lives in tests/valen/export_reference_checks.py and the recipe path is thin entrypoint glue over its main(), so the documented invocation is unchanged. export.md points at the implementation's home and its suite count is refreshed (115 passed, 2 artifact-gated skips).
|
All three items addressed (pushed, new head [P2] importable sourceless modules — addressed in #146 → #147 ordering and the finite-value fixes in #146 — done. #146 is now at Battery relocation — addressed in Re-run at the new head on this host: |
Stacks on #146 (
model/valen-noul-score) — merge #146 first; this PR's diff shows both changes until then, and drops the noul/score commits automatically once #146 merges.Required human contributor answers
Make the pinned Valen checkpoint loadable outside the training-framework pickle, with proof it computes identically.
Numerical parity between the exported artifact set and the original checkpoint through the same kernels.
Every transformation is recorded in a JSON manifest, the LoRA application semantics are anchored, and the parity gate and determinism check are reproducible.
Valen users who need a loadable artifact set with provenance.
After answering the four questions: is a PR needed at all to achieve this outcome?
Yes, without this every consumer converts the pickle on its own.
For a bug fix: link actual observed failure evidence.
N/A.
Purpose
One deterministic command turns the pinned
checkpoint.ptplus its Qwen3.5-2B base into a five-file safetensors artifact set:model.safetensors— the base verbatim, except the trained vision-top blocks 20–23 and merger as plain BF16 replacements, and the 36 FP32 tensors downcast to BF16 (lossless against the reference's BF16 runtime).lora.safetensors— the 372 FP32 LoRA tensors verbatim under their PEFT key names, served unmerged; the manifest'slorablock records the rank 32 / alpha 64 / scale 2 configuration and the application semantics a consumer implements (BF16 base GEMM plus the FP32 delta on the activations, one final rounding — peft 0.18.1Linear.forwardboundaries).head.safetensors— the decision head's two FP32 (256×2048) projections, no bias.config.json— the base config copied verbatim forfrom_pretrained.valen_export.json— the manifest (valen-checkpoint-export/2): per-tensor name/shape/dtype/component/provenance, source revisions and SHA-256s, the downcast list, the head configuration, the consumption path, and the parity record.The export enforces the artifact pins before writing, is deterministic and byte-identical, and validates with streaming digests.
Worker behavior is unchanged; no Rust changes.
Large-change packaging
Added for review readiness (measured at the current head: code/test +4702/−102, total +5210/−143 over 24 files):
tests/valen/export_reference_checks.py); the total text diff is 25 files, +5376/−143. Excluded from the authored count as docs/fixtures/registration: 9 files, +516/−41 (export.md+reference.md, the model README, the two example request JSONs, and the README / supported-models / mkdocs / recipe-README rows). Measured withgit diff --numstat f5ae5acf..7a801f01(merge-base with main at the current head).parityslot and the tests that mutation-check it only have meaning together, so splitting export from verification would land an unverifiable artifact format first and re-review the same manifest code twice; the one independent seam is the Valen: serve noul and score questions through the reference worker #146 noul/score work, which is already split out and merges first.recipe/valen/export_checkpoint.py(deterministic export + manifest schema) → 2.recipe/valen/verify_export_parity.py(dual-stack gate, non-finite rejection) → 3.recipe/valen/compare_reference.py+ its test script (shared comparison + regressions, non-finite scalar rejection) → 4.tests/valen/test_export.py,tests/valen/test_export_parity.py,recipe/valen/test_export_reference.py→ 5. docs and registration, read last.Test Plan
From the repository root with the pinned artifacts (
recipe/valen/reference.mddocuments the setup):Test Result
System1-Omni Version / Commit: head
7a801f01onspec/valen-export(based onmodel/valen-noul-score40f991c, which merges current mainf5ae5acand forward-ports the non-finite gates into #146); the CPU checks were re-run from this committed tree, and the determinism gate (double export, byte-identical) was re-run against the pinned artifacts at70478332— the follow-up commits (80dcc6a3merge,ab7bf996sourceless-stray gate,7a801f01battery relocation) touch only the clean-source gate, its regressions, the battery's location and docs, none of which participates in the export/parity path the determinism and GPU records pin. The GPU parity record is the run pinned inrecipe/valen/reference.mdand the exported manifest's parity slot.Validated on WSL2 x86_64, RTX 5060 Laptop GPU (sm_120), BF16, PyTorch 2.11.0+cu128, peft 0.18.1, pinned Valen source / Preview checkpoint / Qwen3.5-2B base:
pytest tests/valen: 115 passed, 2 artifact-gated skips at the current head (110 + the five clean-source-gate regressions and the shared-gate pin).compileall,test_export_reference.py(with and withoutVALEN_EXPORT, and withoutPYTHONPATH=src, as documented): pass.compare_reference.py --case allon the GPU host: 4/4 cases (image, text, noul, score) PASS at the declared 1e-6 tolerance, re-run at this head after the non-finite scalar gate.Determinism: two consecutive exports produce identical files; manifest
filesdigests —model.safetensors4,548,216,808 B0313882b…,lora.safetensors134,613,824 B4f6d9590…,head.safetensorsf365a09e…,config.jsoned1c1723….Parity: passed with max probability delta 0.0 on both the text and image fixtures — exact answer agreement, equal tokens (48/48 and 108/108), at the declared 1e-6 tolerance. Verbatim block below; recorded in
recipe/valen/reference.mdand the manifest'sparityslot (tool-filled; re-runs refuse to overwrite a different record).Manifest audit: 1006 tensor records, 36 declared downcasts, mutation-tested tamper detection.
Review-follow-up re-verification at
7a801f01(sourceless-stray gate + battery relocation):pytest tests/valen115 passed, 2 artifact-gated skips;test_compare_reference.py35/35 ok;test_export_reference.pyok (2 skips);test_compare_with_backend.pyok;compileallpassed;mkdocs build --strictpassed;pytest tests/benchmarks65 passed. The reviewer's sourceless-module repro (a standalone importablepeft.pycbeside a clean tracked tree) is now rejected by_verify_clean_checkoutbefore any record is produced, and the same gate serves both evidence entrypoints (verify_export_parity.stage_environmentcallsreference._verify_clean_checkout).Not run / unverified: the Rust suite (no Rust changed; CI covers it), any other hardware, and consumption by an actual native Rust loader — the consumption path is documented in the manifest and exercised here through
from_pretrainedplus the peft application; the native worker is separate upcoming work.Demo / evidence
Nonvisual tooling change: video N/A. The evidence is the parity record — verbatim
verify_export_parity.pyoutput at the PR head, also recorded inrecipe/valen/reference.mdwith reproduction commands and the pinned revisions:Usage and artifact documentation:
recipe/valen/export.md; the manifest schema is documented in the tool's docstrings.Self-review