You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While sweeping every gntr slug's assembled gradient against finite differences for #535, one
column on Raia_CancerResearch2011 came back at exactly twice its central difference. It has
not recurred in five subsequent runs at that same point, two of which had byte-identical routing.
I am filing it because a factor of exactly two on a sensitivity column is not the shape numerical
noise takes, and because a non-reproducible wrong gradient is strictly worse than a reproducible
one: the #535 sweep only caught it by chance, and nothing in the product would have.
The observation
Slug Raia_CancerResearch2011 (Grein-2026 subset-I corpus), evaluated at its PEtab nominal point
displaced 2% of each box width along sampling space. ‖∇‖∞ = 2.274e+04, so this is a
well-conditioned point, and the central difference is stable to five digits across three step sizes:
-45475 / -22737.6 = 2.00000. Every other one of Raia's 39 columns agreed to 1e-4 or better on that
same run.
Probing that column directly at the same point, the FD side is sound and the perturbation is
applied exactly (no bound clamping, no silent reflection):
Raia also puts prediction_formula σ on all eight observables (ADR-0079/0080), so the assembled
column runs through the sigma_sensitivity chain rule — the one place where a term could plausibly
be added twice. assembled - truth = -22737, i.e. exactly the true value again, which is consistent
with one contribution being counted twice rather than with a scale factor being wrong. That is a
hypothesis, not a finding; I could not make it happen again to test it.
Not FD noise. Stable to five digits across h ∈ {1e-3, 3e-4, 1e-4}, and forward/backward/
central agree.
Not the evaluation point. The same point at --disp 0.01 and 0.04 agrees to 8e-06 and 1e-06.
Reproduction attempts
Five runs at the identical point since: one in the post-fix sweep, three explicit back-to-back
repeats, and one in the final bounds-clear sweep. All five read -22737 against -22738. The corpus
sweep for #535 now reports Raia clean.
Suspects
No confirmed cause. Recording the one lead: lanl/bngsim#117 established that a module-level printer
in that stack was not thread-safe, and it is what was ultimately behind #525's per-dose sensitivity
mis-stacking. It is closed, and its symptom was a derivative reported as "not representable in
C" rather than a silently scaled sensitivity column, so I want to be clear this is a precedent for
"non-determinism has reached the sensitivity path before", not evidence that it happened here.
Suggested next step
Nothing to fix until it reproduces. The cheap standing guard would be an assertion, or a debug-level
check, that a single-contribution ic-axis route contributes exactly once per scored point — the
narrow invariant this violated. Reproducing it may need the run repeated under load, since every
attempt above was on an otherwise idle machine.
Found during #535; the corpus-side tooling is tools/fd_check.py in wshlavacek/BNGL-Models.
Summary
While sweeping every
gntrslug's assembled gradient against finite differences for #535, onecolumn on
Raia_CancerResearch2011came back at exactly twice its central difference. It hasnot recurred in five subsequent runs at that same point, two of which had byte-identical routing.
I am filing it because a factor of exactly two on a sensitivity column is not the shape numerical
noise takes, and because a non-reproducible wrong gradient is strictly worse than a reproducible
one: the #535 sweep only caught it by chance, and nothing in the product would have.
The observation
Slug
Raia_CancerResearch2011(Grein-2026 subset-I corpus), evaluated at its PEtab nominal pointdisplaced 2% of each box width along sampling space.
‖∇‖∞ = 2.274e+04, so this is awell-conditioned point, and the central difference is stable to five digits across three step sizes:
-45475 / -22737.6 = 2.00000. Every other one of Raia's 39 columns agreed to 1e-4 or better on thatsame run.
Probing that column directly at the same point, the FD side is sound and the perturbation is
applied exactly (no bound clamping, no silent reflection):
Forward, backward and central all agree, so the true derivative is ≈
-22738and the assembledvalue was the wrong one.
Why it is interesting
init_Rec_iis Raia's only initial-condition-axis free parameter. Its route is a singlecontribution and there is nothing in it to double:
Raia also puts
prediction_formulaσ on all eight observables (ADR-0079/0080), so the assembledcolumn runs through the
sigma_sensitivitychain rule — the one place where a term could plausiblybe added twice.
assembled - truth = -22737, i.e. exactly the true value again, which is consistentwith one contribution being counted twice rather than with a scale factor being wrong. That is a
hypothesis, not a finding; I could not make it happen again to test it.
What I ruled out
either side of it — verified by monkeypatching the old rule and diffing
sensitivity_params,sensitivity_ic, and every per-parameter route. Nothing changed, so that fix neither caused norcured this.
set_valueclamps out-of-box values, which makes a centraldifference a half-step and was behind four other red rows in the Verify the assembled gradient against finite differences on every real gntr problem, not just fixtures #535 sweep.
init_Rec_isat atu = 2.211in a[-5, 3]box, nowhere near either bound, and the probe above confirms bothperturbations landed exactly where intended.
h∈ {1e-3, 3e-4, 1e-4}, and forward/backward/central agree.
--disp0.01 and 0.04 agrees to 8e-06 and 1e-06.Reproduction attempts
Five runs at the identical point since: one in the post-fix sweep, three explicit back-to-back
repeats, and one in the final bounds-clear sweep. All five read
-22737against-22738. The corpussweep for #535 now reports Raia clean.
Suspects
No confirmed cause. Recording the one lead: lanl/bngsim#117 established that a module-level printer
in that stack was not thread-safe, and it is what was ultimately behind #525's per-dose sensitivity
mis-stacking. It is closed, and its symptom was a derivative reported as "not representable in
C" rather than a silently scaled sensitivity column, so I want to be clear this is a precedent for
"non-determinism has reached the sensitivity path before", not evidence that it happened here.
Suggested next step
Nothing to fix until it reproduces. The cheap standing guard would be an assertion, or a debug-level
check, that a single-contribution
ic-axis route contributes exactly once per scored point — thenarrow invariant this violated. Reproducing it may need the run repeated under load, since every
attempt above was on an otherwise idle machine.
Found during #535; the corpus-side tooling is
tools/fd_check.pyinwshlavacek/BNGL-Models.