Skip to content

A gradient column assembled at exactly 2x its central difference, once, and has not reproduced (Raia, IC axis) #537

Description

@wshlavacek

Summary

While sweeping every gntr slug's assembled gradient against finite differences for #535, one
column on Raia_CancerResearch2011 came back at exactly twice its central difference. It has
not recurred in five subsequent runs at that same point, two of which had byte-identical routing.

I am filing it because a factor of exactly two on a sensitivity column is not the shape numerical
noise takes, and because a non-reproducible wrong gradient is strictly worse than a reproducible
one: the #535 sweep only caught it by chance, and nothing in the product would have.

The observation

Slug Raia_CancerResearch2011 (Grein-2026 subset-I corpus), evaluated at its PEtab nominal point
displaced 2% of each box width along sampling space. ‖∇‖∞ = 2.274e+04, so this is a
well-conditioned point, and the central difference is stable to five digits across three step sizes:

param            assembled   fd(h=1e-3)  fd(h=3e-4)  fd(h=1e-4)
init_Rec_i         -45475       -22738      -22738      -22739

-45475 / -22737.6 = 2.00000. Every other one of Raia's 39 columns agreed to 1e-4 or better on that
same run.

Probing that column directly at the same point, the FD side is sound and the perturbation is
applied exactly (no bound clamping, no silent reflection):

theta(u0-h) = 162.50845020924407     f(-h) = 2683.0361725734565
theta(u0)   = 162.62074585080055     f(0)  = 2676.2079742098376
theta(u0+h) = 162.73311909023639     f(+h) = 2669.393618322428

forward  diff: -22714.5
backward diff: -22760.7
central  diff: -22737.6

Forward, backward and central all agree, so the true derivative is ≈ -22738 and the assembled
value was the wrong one.

Why it is interesting

init_Rec_i is Raia's only initial-condition-axis free parameter. Its route is a single
contribution and there is nothing in it to double:

seed terms: (SeedTerm(target='ic', key='Rec_i', node=('num', 1.0)),)
route:      [('ic', 'Rec_i', 1.0)]

Raia also puts prediction_formula σ on all eight observables (ADR-0079/0080), so the assembled
column runs through the sigma_sensitivity chain rule — the one place where a term could plausibly
be added twice. assembled - truth = -22737, i.e. exactly the true value again, which is consistent
with one contribution being counted twice rather than with a scale factor being wrong. That is a
hypothesis, not a finding; I could not make it happen again to test it.

What I ruled out

Reproduction attempts

Five runs at the identical point since: one in the post-fix sweep, three explicit back-to-back
repeats, and one in the final bounds-clear sweep. All five read -22737 against -22738. The corpus
sweep for #535 now reports Raia clean.

Suspects

No confirmed cause. Recording the one lead: lanl/bngsim#117 established that a module-level printer
in that stack was not thread-safe, and it is what was ultimately behind #525's per-dose sensitivity
mis-stacking. It is closed, and its symptom was a derivative reported as "not representable in
C" rather than a silently scaled sensitivity column, so I want to be clear this is a precedent for
"non-determinism has reached the sensitivity path before", not evidence that it happened here.

Suggested next step

Nothing to fix until it reproduces. The cheap standing guard would be an assertion, or a debug-level
check, that a single-contribution ic-axis route contributes exactly once per scored point — the
narrow invariant this violated. Reproducing it may need the run repeated under load, since every
attempt above was on an otherwise idle machine.

Found during #535; the corpus-side tooling is tools/fd_check.py in wshlavacek/BNGL-Models.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions