[CLM 2/2] Reach the encoder, answer requests, and check against CLM

docs/supported-models.md

@@ -19,17 +19,20 @@ Models that are being added are also tracked in issues labeled [new model](https
 | Open-Jev-9B | The same [native Rust/CUDA worker](../recipe/open_jev/native.md) | Not supported | Validated on compute capability 8.9 against the reference for [253 requests](../recipe/open_jev/validation-9b.md) | Not supported | Compute capability 8.0 or newer, CUDA toolkit to build, exported merged weights and trained head |
 | autotrust/JEV-27B-VL | [Experimental native worker](../recipe/jev_vl/README.md), single-question API | Contract tests only; no CPU inference | H800 frozen-corpus check; [experimental scope and limits](../recipe/jev_vl/validation.md) | Not supported | Pinned merged checkpoint, rebuilt ABI 6 CUDA library; image inputs require offline Transformers preencoding |
 | CLM-v0.1-8B | [External `clm-serve` recipe](../recipe/clm/README.md) with a CPU stub embeddings server | **Stub-encoder contract checks only** ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)); not real Qwen3-8B decisions | Real encoder unverified by the merged recipe | Unverified | Python, upstream CLM and head checkpoint; a real encoder requires a separate embeddings server |
+| CLM-v0.1-8B | [Native Rust engine](../src/models/clm/README.md), `clm-run`, in front of a Qwen3-8B `/v1/embeddings` server | Engine-only agreement **4.5e-06** on identical vectors ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)); contract checks documented ([#23](https://github.com/ThinkFlowLab/system1-omni/pull/23)) | **Validated on compute capability 8.9** ([#29](https://github.com/ThinkFlowLab/system1-omni/pull/29), [validation](../recipe/clm/native/VALIDATION.md)) | Unverified | Rust to build and a separate Qwen3-8B `/v1/embeddings` server for the encoder; the engine itself has no device-specific code |
 | OmniJev-4B v1.1 | None yet: [checkpoint export and CPU contract](../recipe/omnijev/README.md); the native worker is in progress ([#114](https://github.com/ThinkFlowLab/system1-omni/issues/114)) | Request contract, preparation, heads and finishing checked against the reference ([#120](https://github.com/ThinkFlowLab/system1-omni/pull/120)); no inference | Not yet | Not supported | Python 3.10 reference environment for the export and fixtures |
 
 - **Validated:** covered by the recipe on `main` or by the checks in the linked merged pull request.
 - **Unverified:** the worker accepts this device, but no recipe or merged pull request covers it.
 - **Planned:** not implemented yet; the linked issue tracks it.
 
 The Cua-S1 workers answer `choice` questions only.
-LAYA's English worker and Open-Jev support `choice`, `score`, and `noul` text questions.
-CLM's merged recipe exercises these answer shapes with stub embeddings; it does
-not validate decision quality. MPS validation above is for a Python/PyTorch
-worker, not a native Metal backend.
+LAYA's English worker, Open-Jev and CLM support `choice`, `score`, and `noul` text questions.
+CLM has a row per path: the merged `clm-serve` recipe exercises those answer shapes with stub
+embeddings and says nothing about decision quality, while the native engine's CUDA row is the
+path through a real Qwen3-8B encoder. Its engine-only figure comes from the same comparison run
+against the deterministic stub, where both sides get identical vectors; neither number validates
+decision quality. MPS validation above is for a Python/PyTorch worker, not a native Metal backend.
 
 The [architecture contracts](architecture.md) describe the native target.
 Shared processing orchestration and dynamic batching remain planned. Native

recipe/clm/README.md

@@ -90,3 +90,7 @@ GPU=0 PORT=8090 UTIL=0.35 ./serve_qwen3_8b.sh     # from the CLM checkout; needs
 ```
 
 Everything downstream is unchanged, which is the property this recipe is meant to demonstrate.
+
+`recipe/clm/native/VALIDATION.md` records what that path measures when it is run against the
+native engine instead of `clm-serve`: the engine alone on identical vectors, and the whole
+path on a real Qwen3-8B.

recipe/clm/native/VALIDATION.md

@@ -0,0 +1,121 @@
+# CLM CUDA validation
+
+`omni-clm` owns everything after the encoder: a frozen Qwen3-8B runs as its own process
+behind an `/v1/embeddings` endpoint, and the engine owns the two projection heads, the
+cosine score, the temperature and the typed answer. Agreement with the reference is
+therefore two separate questions — whether the engine's own arithmetic matches, and whether
+the whole path still agrees once a real encoder is in front of it — and they are measured
+separately below, because one number cannot answer both.
+
+Both phases ran on one RTX 4090 on 2026-10-06. Every number below is the output of
+`compare_with_reference.py`, which sends the same requests to `clm-run` and to CLM's own
+`Engine` and `Schema` and compares the answers.
+
+## The environment
+
+| | |
+| --- | --- |
+| GPU | NVIDIA GeForce RTX 4090, 24 GB, compute capability 8.9 |
+| Driver | 595.71.05 |
+| CUDA | 13.0, V13.0.88 |
+| Python | 3.12.3 |
+| torch | 2.12.1+cu130 |
+| transformers | 5.17.0, the encoder |
+| contrastive-lm | 0.1.0, the reference |
+| Rust | 1.98.1 |
+| Engine commit | `a2019f9` |
+| `clm-run` | sha256 `791859b51047edd79ad6f380bba9a78ea1fdddb5c4a8ad797472ac541251dc92` |
+
+## Phase A: the engine alone
+
+`recipe/clm/stub_embedder.py` derives its vectors from the text alone, and both sides ask
+it, so the same text reaches both as the same vector. The encoder's numerical difference is
+out of the comparison; what remains is parsing, the text the heads see, the projections, the
+cosine, the temperature and the answer shape — and, because a difference in the text the two
+sides send would produce a different vector, the text is in the comparison too.
+
+| case | kind | agreement |
+| --- | --- | --- |
+| `choice_two` | choice | `max\|dp\|=3.12e-07`, both pick `billing` |
+| `choice_five` | choice | `max\|dp\|=4.49e-06`, both pick `c` |
+| `score_three` | score | `max\|dp\|=1.13e-06`, `1.132559` vs `1.132558` |
+| `noul_stmt` | noul | `0.271456` vs `0.271451` |
+| `noul_numbers` | noul | `0.225941` vs `0.225943` |
+
+Worst case **4.49e-06**. Nothing here is passed a tolerance to hide behind: the answers are
+compared at the 0.025 the script uses for the real encoder, and they land more than three
+orders of magnitude inside it.
+
+## Phase B: end to end, on Qwen3-8B
+
+`recipe/clm/native/transformers_encoder.py` serves Qwen3-8B — the model the deployment
+encodes with — behind the same endpoint, so vLLM is not needed to check the client.
+
+| case | kind | agreement |
+| --- | --- | --- |
+| `choice_two` | choice | `max\|dp\|=1.72e-05`, both pick `billing` |
+| `choice_five` | choice | `max\|dp\|=1.97e-02`, both pick `a` |
+| `score_three` | score | `max\|dp\|=1.08e-02`, `0.385166` vs `0.403241` |
+| `noul_stmt` | noul | `0.631038` vs `0.646805` |
+| `noul_numbers` | noul | `0.323871` vs `0.321280` |
+
+Worst case **1.97e-02**, against the script's tolerance of 0.025, and all five cases agree.
+
+## Why the two numbers differ, and why the tolerance is 0.025
+
+CLM scores with `softmax(exp(logit_scale) * cos(...) / temperature)`, and the published
+checkpoint's `logit_scale` is 4.6132, whose exponential the engine caps at 100. A difference
+of `d` in a cosine therefore becomes `100 d` in the logit, and two forward passes that
+disagree by ~1e-4 in the cosine — which is what two encoder implementations produce for
+Qwen3-8B — disagree by ~1e-2 in the probabilities.
+
+Phase A is what makes that attribution a measurement rather than a story: with the encoder
+taken out, the engine agrees to 4.49e-06, so the ~1e-2 in phase B is the encoder and not the
+engine. A deployment serving the same Qwen3-8B weights would be on the other side of that
+difference, which is a property of how this model scores, not a defect on either side.
+
+## Artifacts
+
+| file | sha256 |
+| --- | --- |
+| `CLM_v0.1-8B.pt` | `b2b4a8c9c2d39263eff78a351eb909a342ce9b3bf21a3f07c1d1bf15f1c4eda5` |
+| `clm-export/oracle.json` | `7d058de4a691b6d4949aa51c53c7ec95a136ac4c14290580e2d81ba8f7192c16` |
+| Qwen3-8B `config.json` | `f7c4eadfbbf522470667b797a3c89be2524832d2d599797248dc304fff447c30` |
+
+`clm-export/model.safetensors` is deliberately not pinned. The safetensors serializer
+carries `__metadata__` through a hash map, so its key order — and with it the file hash —
+differs from run to run: four exports of this same checkpoint gave four file hashes while
+their tensor entries were byte-identical and every one of them wrote the same `oracle.json`.
+`oracle.json` holds the FP32/FP16/BF16 hash of every tensor and is stable, byte-identical
+across two machines, so that is what pins the conversion.
+
+## Reproduction
+
+Both phases take the same comparison; only the endpoint changes.
+
+```sh
+# the checkpoint, once
+python recipe/clm/native/export_weights.py CLM_v0.1-8B.pt clm-export
+
+# Phase A, no GPU and no 8B encoder
+python recipe/clm/stub_embedder.py --port 8090 &
+
+# Phase B, instead of the stub, on a GPU
+python recipe/clm/native/transformers_encoder.py --model /path/to/Qwen3-8B --port 8090 &
+
+python recipe/clm/native/compare_with_reference.py \
+  --checkpoint clm-export --bin target/release/clm-run \
+  --emb-url http://127.0.0.1:8090/v1/embeddings \
+  --pt CLM_v0.1-8B.pt --temperature 1.0
+```
+
+## What this does not establish
+
+- **Decision quality.** Five fixed requests show the port is faithful; they are not a
+  labelled set and say nothing about how often CLM is right.
+- **vLLM.** The encoder here is Transformers. A deployment's pooling server is a different
+  implementation and sits on the other side of the ~1e-2 above; nothing here measures it.
+- **Latency or throughput.** Nothing was timed, and there is no candidate-vector cache to
+  exercise: `Engine::decide` embeds every text on every call.
+- **Other hardware.** One RTX 4090, compute capability 8.9. The engine is Rust and has no
+  device-specific code; the CUDA part of this path is `transformers_encoder.py`.

recipe/clm/native/compare_with_reference.py

@@ -0,0 +1,154 @@
+"""Compare omni-clm's decision with the CLM reference, over the same embeddings.
+
+Both sides call the same OpenAI-compatible `/v1/embeddings` endpoint, so the vectors are
+identical and the only difference left is the code between the vectors and the answer.
+
+    python recipe/clm/native/compare_with_reference.py \
+        --checkpoint /path/to/clm-export \
+        --bin /path/to/clm-run \
+        --emb-url http://127.0.0.1:8090/v1/embeddings \
+        --pt /path/to/CLM_v0.1-8B.pt
+
+## On the tolerance
+
+CLM scores with `exp(logit_scale) = 100`, so a difference of `d` in a cosine similarity
+becomes `100 d` in the logit. Two encoder forward passes that disagree by ~1e-4 in the
+cosine therefore disagree by ~1e-2 in the probabilities, and that is what two different
+implementations of Qwen3-8B produce -- vLLM's bf16 pooling against Transformers' forward.
+It is a property of how this model scores, not a defect on either side.
+
+The default tolerance reflects it. The engine's own arithmetic is checked separately and
+much more tightly: with the vectors held fixed the two implementations agree to 3e-06.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+CASES = [
+    ("choice_two", "choice", {"billing": "Charges and refunds", "technical": "Software problems"}),
+    ("choice_five", "choice", {"a": "alpha", "b": "beta", "c": "gamma", "d": "delta", "e": "epsilon"}),
+    ("score_three", "score", ["Not urgent", "Needs attention soon", "Needs attention immediately"]),
+    ("noul_stmt", "noul", None),
+    # A `noul` reads its two descriptions in `false`/`true` order, not in the order they
+    # were written, so a number here has to be found by the key it was written under.
+    # Counting positions gave `false: 2` / `true: 18446744073709551616` reversed, which
+    # is a different pair of candidate texts and so a different answer.
+    ("noul_numbers", "noul", {"true": 18446744073709551616, "false": 2}),
+]
+
+STATE = "I was charged twice for order 4411 and want the second charge refunded."
+
+# exp(logit_scale) with the published checkpoint, and what it does to a cosine difference.
+SCALE = 100.0
+TOLERANCE = 2.5e-2
+
+
+def request_for(name: str, kind: str, criteria) -> dict:
+    q = {"type": kind, "instructions": f"Question {name}"}
+    if criteria is not None:
+        q["criteria"] = criteria
+    return {"model": "clm-latest", "state": STATE, "questions": {name: q}}
+
+
+def reference(emb_url: str, emb_model: str, pt: Path, req: dict, temperature: float) -> dict:
+    """CLM's own heads and schema, applied to the same embeddings the Rust side gets."""
+    from clm.embedder import Embedder
+    from clm.engine import Engine
+
+    embedder = Embedder(url=emb_url, model=emb_model)
+    engine = Engine(embedder=embedder, checkpoint=str(pt))
+    return engine.answer(req["state"], req["questions"], "clm-latest", temperature)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--checkpoint", type=Path, required=True, help="converted export dir")
+    parser.add_argument("--bin", type=Path, required=True, help="clm-run")
+    parser.add_argument("--emb-url", required=True)
+    parser.add_argument("--emb-model", default="qwen3-8b")
+    parser.add_argument("--pt", type=Path, required=True, help="CLM_v0.1-8B.pt")
+    parser.add_argument("--temperature", type=float, default=1.0)
+    parser.add_argument(
+        "--tolerance",
+        type=float,
+        default=TOLERANCE,
+        help=f"logits are scale*cos with scale={SCALE:g}, so two encoder implementations "
+        f"differ by about this much in the probabilities",
+    )
+    args = parser.parse_args()
+
+    requests_ = [(n, request_for(n, k, c)) for n, k, c in CASES]
+    payload = "".join(json.dumps(r) + "\n" for _, r in requests_)
+    proc = subprocess.run(
+        [
+            str(args.bin),
+            str(args.checkpoint),
+            "--emb-url",
+            args.emb_url,
+            "--model",
+            args.emb_model,
+            # Both sides must run the same temperature, or a non-default one compares two
+            # different configurations and reports a port regression that is not there.
+            "--temperature",
+            str(args.temperature),
+        ],
+        input=payload,
+        capture_output=True,
+        text=True,
+        timeout=600,
+    )
+    if proc.returncode != 0:
+        print(proc.stderr[-2000:], file=sys.stderr)
+        raise SystemExit(f"clm-run exited {proc.returncode}")
+    mine = [json.loads(line) for line in proc.stdout.splitlines() if line.strip()]
+    if len(mine) != len(requests_):
+        raise SystemExit(f"clm-run answered {len(mine)} of {len(requests_)}")
+
+    failures = 0
+    for (name, req), got in zip(requests_, mine):
+        want = reference(args.emb_url, args.emb_model, args.pt, req, args.temperature)
+        mine_p = got["answers"][name]
+        want_p = want["answers"][name]
+        kind = req["questions"][name]["type"]
+
+        if kind == "noul":
+            got_v, want_v = mine_p["noul"], want_p["noul"]
+            ok = abs(got_v - want_v) <= args.tolerance
+            label, detail = "noul", f"{got_v:.6f} vs {want_v:.6f}"
+        else:
+            gk, wk = mine_p["probabilities"], want_p["probabilities"]
+            if set(gk) != set(wk):
+                print(f"FAIL {name}: keys differ {sorted(gk)} vs {sorted(wk)}")
+                failures += 1
+                continue
+            worst = max(abs(gk[k] - wk[k]) for k in gk)
+            same_pick = (
+                mine_p.get("choice") == want_p.get("choice")
+                if kind == "choice"
+                else abs(mine_p["score"] - want_p["score"]) <= args.tolerance
+            )
+            ok = worst <= args.tolerance and same_pick
+            label = "choice" if kind == "choice" else "score"
+            detail = f"max|dp|={worst:.2e} " + (
+                f"choice={mine_p['choice']}/{want_p['choice']}"
+                if kind == "choice"
+                else f"score={mine_p['score']:.6f}/{want_p['score']:.6f}"
+            )
+        failures += not ok
+        print(f"{'PASS' if ok else 'FAIL'} {name:12} {label:7} {detail}")
+
+    print(
+        f"\n{'all cases agree' if not failures else str(failures) + ' FAILED'} "
+        f"(tolerance {args.tolerance:g}; point --emb-url at recipe/clm/stub_embedder.py "
+        f"to measure the engine alone, on identical vectors)"
+    )
+    raise SystemExit(1 if failures else 0)
+
+
+if __name__ == "__main__":
+    main()