Skip to content

fix(delta): persist raw_imports in a deterministic order (follow-up to #3227) - #3234

Merged
squid-protocol merged 2 commits into
mainfrom
fix/3227-raw-imports-ordering
Sep 20, 2026
Merged

squid-protocol merged 2 commits into
mainfrom
fix/3227-raw-imports-ordering

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Follow-up to #3227 / #3220. Small, self-contained.

The problem

#3220 persisted raw_imports as json.dumps(sorted(...)). sorted() over the mixed str / (module, entity) list raised TypeError: '<' not supported between instances of 'tuple' and 'str', and #3227 fixed it by dropping the sort entirely, with the comment "Order is irrelevant -- it round-trips into a set."

The round-trip is fine. The order isn't irrelevant.

A rehydrated file's raw_imports is a set (state_rehydrator._json_import_set; test_state_rehydrator.py:132 already asserts isinstance(..., set)), so persisting it in iteration order makes the stored column depend on PYTHONHASHSEED. Measured on the exact expression at record_keeper.py:1333:

seed 0:     ["m/n.py", "missing_pkg", "src/lib.py", "z/y.py", ["src/lib.py","helper"], "src/util.py", "a/b.py"]
seed 1:     ["m/n.py", "missing_pkg", ["src/lib.py","helper"], "a/b.py", "src/util.py", "z/y.py", "src/lib.py"]
seed 2:     ["src/util.py", ["src/lib.py","helper"], "src/lib.py", "missing_pkg", "z/y.py", "m/n.py", "a/b.py"]
seed 12345: ["missing_pkg", ["src/lib.py","helper"], "src/lib.py", "m/n.py", "src/util.py", "z/y.py", "a/b.py"]

Why it matters here specifically

That order is the order the resolver walks a file's imports → the order _publish_edges emits rows → the order #3183 assigns edge_data AUTOINCREMENT ids in.

To be precise about the blast radius, because it is narrower than it first looks:

  • Edge values are safe. weight, import_statements and entity_imports accumulate with += in build_dependency_graph, and addition is commutative. No number changes.
  • Row order and ids are not safe. A delta scan can produce the same edges as a full scan in a different order, with different ids — in the PR whose entire purpose is delta parity.

It is also the bug class this repo just swept in #3212/#3223 ("sort every set you serialise"; unresolved_calls = list(set) was one of them).

The fix

def _ordered_raw_imports(raw_imports: Any) -> list:
    return sorted(
        (list(x) if isinstance(x, tuple) else x for x in (raw_imports or [])),
        key=lambda x: (1, x) if isinstance(x, list) else (0, [x]),
    )

Two sort groups, so a str is never compared to a tuple — which is exactly what broke the original sorted(). Verified stable across PYTHONHASHSEED 0/1/2/99/12345 (five seeds → one distinct output).

Scope note worth recording

No production path emits a tuple today. galaxyscope.py:734 builds raw_imports from raw_imports.add(clean_module) — strings only — and both it and :2048 would themselves raise on a tuple (sorted(), then .strip()). The (module, entity) shape is live only in network_risk_sensor's documented "Level 2 Tuple (Entity Import)" branch and in the test_edge_data.py fixture.

So the TypeError #3227 hit was reachable from the fixture, not from a scan. The tuple handling stays as defensive support for the sensor's API; the reachable bug is the ordering one, over a rehydrated set of plain strings. Flagging it in case the intended end state is for tuples to reach ram_cache — if so, those two galaxyscope.py call sites need the same treatment and that's a separate issue.

Verification

pytest tests/core_engine tests/tools_recorders    1530 passed, 4 skipped, 1 xfailed
  (incl. the 7 that failed in #3227: test_edge_data + test_uncomputed_graph_metrics)
audit_check.py                                    all clear
PYTHONHASHSEED 0/1/2/99/12345                     5 seeds -> 1 distinct output

Three regression tests in test_state_rehydrator.py: ordering is stable over 50 reshuffles of the same set, the ordering round-trips back through _json_import_set with tuples restored as tuples, and the empty/None cases.

🤖 Generated with Claude Code

…#3227)

#3220 persisted raw_imports with `json.dumps(sorted(...))`. That raised
TypeError('<' not supported between tuple and str) on the mixed str/(module,
entity) list, and #3227 fixed it by dropping the sort entirely.

The order is not irrelevant. A rehydrated file's raw_imports is a set
(state_rehydrator._json_import_set; test_state_rehydrator.py:132 asserts it), so
persisting it in iteration order makes the column hash-seed dependent:

    seed 0:     ["m/n.py", "missing_pkg", "src/lib.py", "z/y.py", ...]
    seed 1:     ["m/n.py", "missing_pkg", ["src/lib.py","helper"], "a/b.py", ...]
    seed 2:     ["src/util.py", ["src/lib.py","helper"], "src/lib.py", ...]

That order is the order the resolver walks the imports, which is the order
`_publish_edges` emits rows, which is the order #3183 assigns AUTOINCREMENT ids
in. Edge VALUES are safe -- weight/import_statements/entity_imports accumulate
with `+=`, and addition is commutative -- but row order and ids are not, in a
change whose purpose is delta parity.

`_ordered_raw_imports` sorts in two groups so a str is never compared to a
tuple, which is what broke the original sorted(). Verified stable across
PYTHONHASHSEED 0/1/2/99/12345.

Scope note: no production path actually emits a tuple today. galaxyscope.py:734
builds raw_imports from `raw_imports.add(clean_module)` (strings only), and both
it and :2048 would themselves raise on a tuple. The (module, entity) shape is
live only in network_risk_sensor's "Level 2" branch and the test_edge_data
fixture, so the tuple handling stays as defensive support -- the reachable bug
is the ordering one, over a rehydrated set of plain strings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment thread tests/core_engine/test_state_rehydrator.py Fixed
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

CodeQL note: `gitgalaxy.core.state_rehydrator` was imported both as a module
(`import ... as sr`) and with `from ... import StateRehydrator`. Only one symbol
was needed from the module form, so it joins the existing `from` import and the
alias goes away entirely -- rather than the suggested
`StateRehydrator = sr.StateRehydrator` rebinding, which keeps both import forms
and adds a shadowing assignment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@squid-protocol
squid-protocol merged commit 66d224f into main Sep 20, 2026
29 checks passed
@squid-protocol
squid-protocol deleted the fix/3227-raw-imports-ordering branch September 20, 2026 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants