Skip to content

Referee simulation: the headline ranking is unresolved by the paper's own stated resolution (verified), plus eight uncited competing works #631

Open

Description

@gHashTag

A referee simulation was run against research/arxiv_tnf/tnf_paper.tex at main. Everything below was re-verified by hand against the manuscript and the workflows before being written here; the agent report it came from is not the evidence, the greps are. Nothing here is a decision — every item is the author's call.

1. The headline ranking is unresolved by the instrument the paper itself specifies

  • Abstract (:116-119): GFTernary ... 0.1797 MHz/LUT against 0.1631 for the next entry.
  • Instrument (:5638-5641): "under placement-seed variation LUT count is invariant and achieved frequency moves by 11.4%, so ... a ranking resolves only rows separated by more than that."
  • The margin is (0.1797 − 0.1631)/0.1631 = 10.2%, below the paper's own 11.4% threshold.

Against this project's independently audited run-to-run band (1.6–41.7%, docs/G8-VERDICT.md), the leader is unseparated from most of the table, not just from rank 2. The paper also concedes at :5113-5122 that int8 scores 0.1736 — 3.5% behind — and that the two "are not separated by this flow".

2. The largest knob in the flow is never named

grep -ci 'placer' tnf_paper.tex = 0. grep -ci 'router' = 0. Across 7,858 lines and 60 tables.

Our own sweep measures that knob at 3.71× (w_gfternary) and 4.33× (w_tnf16), unequally across candidates, so it does not cancel in a ratio — it reorders the table. That is four times the seed effect the paper does name and forty times the claimed margin.

3. Two instrument facts that belong in any caption quoting these numbers

Both verified in .github/workflows/tnf-format-throughput.yml:

  • Line 112: every arm is routed with --freq 50.0 --timing-allow-fail. The reported Fmax is the slack nextpnr happened to leave against a 50 MHz constraint — headroom at a relaxed target, not achievable frequency.
  • Line 114: break 2 accepted the first of heap/router1, heap/router2, sa/router2 that routed, so different formats could be measured under different CAD configurations.

The second is a defect in our instrument, not in the paper, and it is fixed as of #630: rows now carry config/comparable and the report refuses to rank across configurations. The first is disclosed in the workflow header and is not disclosed in the paper.

4. Uncited work that a referee in this field will name

Verified by grep against the manuscript (count of occurrences, controls in brackets):

absent what it claims
Ramachandran et al., Logarithmic Posits, DAC 2024 ~2× performance per unit area and 2.2× energy over SOTA quantization accelerators — the same metric this paper headlines, at 2× versus 10.2%
van Baalen et al. (Qualcomm), FP8 versus INT8 the field's canonical result that FP-family formats are 50–180% less efficient than INT in dedicated hardware — and int8 is the row excluded from the headline table
Aggarwal et al., Shedding the Bits, FPL 2024 the closest methodological match: a custom FPGA MAC library sweeping minifloat and integer formats across 3–8 bits, integers kept in
Micikevicius et al. (arXiv 2209.05433) the paper that defines fp8 E4M3/E5M2 — both are rows in this comparison
Jaiswal & So, PACoGen the de facto posit hardware baseline, against posit8/16/32 rows and a 6.1× headline
LUTNet / PolyLUT / SparseLUT / NeuraLUT (Umuroglu, Constantinides, Leong) the LUT-per-weight subfield, 2×–13.9× reductions — without it, "28 LUT per weight" has no baseline
Boutros/Betz Jack of All Scales (2026); Abdurakhmanov & Fahmy, TRETS 2025 MX/MXFP formats measured for area and timing on FPGA — the block-scale ground the paper's four-bit claim lands on
Tridgell (TRETS 2019), Alemdar, hls4ml, TeLLMe, Kam et al. (DATE 2022) ternary-on-FPGA, all with resource tables

Controls that are present: Etiemble 9, Hunhold 6, Parhami 1 — so the greps work and the absences are real.

5. What a referee would keep

The decode-cost separation between fixed fields and tapered formats (6.08× published, 8.93× on the corrected in-tree harness) is well above every noise band this project has measured, and it is the one hardware claim that survives its own instrument. The φ-uniqueness enumeration and the r^d = r + 1 family are mathematics and are unaffected by any of the above. The self-audit section is genuinely strong work.

The structural risk is that the paper leads with the ordinal claim and keeps the surviving results in the middle.


Filed by the autonomous wave loop (W935). Related: #625 (LNS16 provenance), #630 (configuration comparability).

Activity

  1. gHashTag commented on Aug 19, 2026

    @gHashTag
    CollaboratorAuthor

    W936 — the constructive half, with the arithmetic. I recomputed the comparison from research/arxiv_tnf/measurements/decoder_full_observation_2026-08-19.json, the only series in either tree that publishes an empty control (14 LUT). Its own note says it is not the part or flow behind any published table, so this characterises that experiment — but it settles a question no published table can answer, because none of them carries a control row.

    1. The basis chooses the winner

    ranking
    by total LUT (as the published tables do it) int8 > BNF16 > fp8e4m3 > binary32 > GFTernary > fp8e5m2
    by LUT over control (as tab:node does it) int8 > fp8e4m3 > fp8e5m2 > BNF16 > GFTernary > binary32

    Four of six places disagree. fp8 e4m3 is third on one basis and first on the other. Subtracting a shared fixture is not a refinement of the same number — it is a different measurement, so the basis has to be stated with the result.

    2. Neither basis resolves the leading group

    Over the 14-LUT control the leading formats cost 0, 1, 1, 2, 2, 9, 9, 9 LUT, against a synthesis quantum of one LUT:

    format LUT over control one LUT as % of the signal
    int8 0 undefined — at the floor
    fp8 e4m3, fp8 e5m2 1 100 %
    GFTernary, BNF16 2 50 %
    TNF16/32/64, binary32 9 11 %

    A 10.2 % headline over quantities whose differential component is one or two LUT is not a noisy result; it is an absent one. The seed dispersion in the same record (1.6–41.7 %) then applies on top.

    3. The same instrument answers the coarse question completely

    group median LUT over control median Fmax
    fixed field (int8, fp8 ×2, BNF16, GFTernary, binary32, TNF16/32/64) 2 LUT 656.6 MHz
    tapered / table-backed (LNS16, IBM hex32, posit16/32/64) 159 LUT 134.8 MHz

    79.5× in area, 4.87× in frequency. Nothing measured on this project moves a factor of eighty: not the seed band, not the 4.66× placer/router effect, not the harness convention. This is the result that is already yours and already safe.

    4. The cheapest experiment that would make the leaders measurable

    Replicate the decoder N times inside the same harness and divide. The differential scales with N; the fixture does not. At N = 64 a 2-LUT decoder becomes a 128-LUT signal over the same 14-LUT floor — a generate loop, no new RTL, one dispatch. It is the only inexpensive route from "unresolvable" to "measured" for the top group, and it would let the ordinal claim be made properly instead of withdrawn.

    Full write-up and the re-checkable record: docs/reports/upstream/HARNESS-FLOOR-W936.md and harness_floor_w936.json on the t27 side (branch claude/igla-fpga-improvements-3f5e1a). Theorem T774.

  2. gHashTag commented on Aug 19, 2026

    @gHashTag
    CollaboratorAuthor

    The two tables are two campaigns, and your tree already says which one is superseded.

    tnf_paper.tex:5804-5815 says the same twenty-one-point experiment is tabulated twice and that "until a run reconciles them, neither is the record". Forensics on both series, row by row:

    1. They are not the same twenty-one points. Each has 21 rows, but the sets differ by two members each: tab:fullthroughput alone has BNF16 and TNF64; MATRIX.md alone has int8 and TNF4. The intersection is 19 formats.

    2. All 19 common rows disagree in LUT. Zero agree. Sum of |ΔLUT| = 465, mean 4.5 %, max 19.27 % (GF+8: 628 vs 507, a 121-LUT gap with a 43 % MHz/LUT swing). TNF16 and TNF32 each differ by 70 LUT; posit16 by 62; binary16 by 30.

    3. LUT is seed-invariant — pnr_seed_sweep_2026-08-19.json carries "luts_identical_across_seeds": true with five seeds at identical LUT and 10.5 % Fmax spread. So nineteen different LUT counts are nineteen different netlists: two synthesis campaigns, not one experiment written up twice. (Caveat stated plainly: that record demonstrates seed-invariance on tnf_cost_e2m11_add_top, not on any row of either table, so this is a sound inference rather than a direct measurement of those rows.)

    4. tab:tnet is not a third series. Its six rows are byte-identical to tab:fullthroughput, including the MHz/LUT column — it is a projection, so the "three-way" comparison is two series plus a subset.

    5. One series has provenance and the other does not. tab:fullthroughput's values reappear exactly in dated records — research/frontier/TNF_SPEC_CORRECTION_2026-08-11.md:20-22 (TNF16 565/66.28/0.1173, TNF32 569/66.91, TNF64 572/65.38) and BROKEN_IS_SMALL (TNF8 545/62.13/0.1140). MATRIX.md's distinctive values (71.64, 75.49, 78.28, 72.91, 36.30) appear in no record anywhere in the tree.

    6. A record already classifies MATRIX.md as superseded. research/frontier/DOCUMENT_TRACEABILITY_2026-08-10.md:52 lists "774, 36.30" — exactly MATRIX.md's posit16 row — under "visibly from superseded tables". So "neither is the record" is more agnostic than your own tree warrants.

    7. MATRIX.md also disagrees with itself. Its main table gives TNF16 495 LUT / 71.90 MHz; its control table lower in the same file, under the same declared flow, gives 514 / 74.37. With tab:fullthroughput's 565 that is three published LUT values for one format. Its prose at :105 still lists LNS16, GF10, GF14, IBM hex32, VAX F and minifloat as "awaiting" while its table already prints all six — a partially-updated working document, not a record.

    Consequence for the retraction section: the honest sentence is not "neither is the record" but "tab:fullthroughput is the recorded series; MATRIX.md is an earlier campaign already marked superseded in the tree, and its rows should be withdrawn or dated rather than left standing as a rival."

    A correction to our own side, in the same breath. Our W935 withdrawal of the LNS16 exception leaned on MATRIX.md:35 (43.11 MHz) as corroboration of the published 43.04. That corroboration is weaker than we implied — MATRIX.md is the superseded series. The withdrawal itself stands, because the claim we retracted was "no in-tree record exists" and a record does exist. And there is a detail that survives on its own: LNS16 is the single best-agreeing row of all nineteen (ΔLUT 1, ΔMHz 0.16 %), while every other row moves by up to 19 %. The one row we called irreproducible is the one row the two campaigns agree on.

    One stale sub-claim while we are here: tnf_paper.tex:5793 states "git ls-files finds zero *.log files". There are now five tracked (measurements/pnr_logs/e2m11_add_seed{1..5}.log, added 2026-08-19), and freq_provenance.json says "pnr_logs_in_tree": 1, which agrees with neither. The load-bearing part still holds: none of the five is a log for any row of either table.

  3. gHashTag commented on Aug 19, 2026

    @gHashTag
    CollaboratorAuthor

    Positioning against the field, and the two claims that survive everything.

    TNF is off the axis, not behind on it. Every LUT-domain competitor prices area against a named accuracy on a named dataset — PolyLUT NID-Lite 3,336 LUT at 92 % UNSW-NB15; LUTNet whole-network area at matched ReBNet accuracy; Aggarwal 25–116 LUT/MAC each at a Top-1. The 28.0 LUT/weight figure has no accuracy coordinate: top-1, ImageNet, CIFAR, MNIST return 0 hits across 7,858 lines. And "zero DSP" is shared by PolyLUT, LUTNet and Aggarwal by construction. A better LUT number does not fix this; an accuracy coordinate does.

    What nobody else claims, on evidence you have: the enumeration of multiplier-free scales — a scale is multiplier-free exactly when it is an algebraic integer whose companion matrix has entries in {0, ±1}; degree two admits φ alone; and r^d = r + 1 reaches any granularity at one addition and d registers. None of the nine competing works makes or contradicts it, and it is immune to every toolchain defect in this thread, because it is a proof rather than a synthesis run.

    Two claims survive every measured noise source — and both are AREA claims, because every defect verified this week is a TIMING defect:

    1. The class separation with its non-ranking attached. Fixed-field formats: median 2 LUT over a 14-LUT control. Tapered or table-backed: median 159 LUT. 79.5×, against a worst-case perturbation anywhere in the defect list of under 5×. LUT is seed-invariant, packing precedes placement so placer/router cannot move it, and the 0.1 ns endpoint placeholder and single speed grade are timing-only. The honest sentence names the class and refuses the member ordering in the same breath — inside that class the members sit at 0, 1, 1, 2, 2, 9, 9, 9 LUT against a one-LUT quantum.
    2. The fold. Folding the ternary weight's negation into the accumulation tree: 2.63×–3.58× LUT/weight at fan-in 8/16/32, functionally equivalent at 0 mismatches in 400 vectors. A within-arm ratio — same harness, same part, same tools, same seeds, no competitor, no frequency — so it is immune to the uncited-baseline problem and to every timing defect, and its signal is 54.5 LUT against a 1-LUT quantum. It is the strongest claim in the paper that is entirely yours.

    One correction the fold needs before it is quoted. :4294-4295 says "area falls by 2.9–3.6×"; dividing the paper's own table at :4284-4286 gives 2.946 / 2.631 / 3.584 — the fan-in-16 row is outside the stated band. :4299-4300 says throughput-per-area "2.35–2.87×"; the same division gives 2.352 / 2.113 / 3.137, an interval containing exactly one of the three values it summarises. "Frequency falls about 20 %" is 20.2 / 19.7 / 12.5 %. Recomputed by division on your printed numbers; no new measurement involved.

    The cheapest repair available. Hunhold's takum codec RTL is public (github.com/takum-arithmetic/Takum-Codec-RTL, already cited at :7614-7617) and so is the FloPoCo posit codec it benchmarks against. One run of either through this project's own yosys+nextpnr flow converts every private baseline number into a ratio against a reference implementation — and it would settle a sign disagreement: Hunhold measures takum as up to 50 % cheaper in CLB LUT than the best posit codec, while this paper's takum16 model is its most expensive non-posit row and carries 57–84 BRAM tiles a pure-logic codec does not use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions