Skip to content

Fix #825: Extraction hardening for scala - #898

Merged
squid-protocol merged 1 commit into
mainfrom
feature/825-extraction-hardening-scala
Jul 31, 2026
Merged

squid-protocol merged 1 commit into
mainfrom
feature/825-extraction-hardening-scala

Conversation

@squid-protocol

@squid-protocol squid-protocol commented Jul 31, 2026 •

Copy link
Copy Markdown
Owner

Summary

Part of the extraction-hardening epic (#813). Closes #825.

Hardens all four extraction gauntlets (func_start, args, class_start, _dependency_capture) for scala per tests/extraction/how_to_harden_extraction.md. This was a fresh sweep (no pre-flagged known bug) and surfaced three real bugs:

  • func_start/args shared a flat \[[^\]]*\] Rule-11 square-bracket generic-parameter step-over, breaking any nested generic bound (def foo[T <: Comparable[T]](x: T): T = {, a realistic bounded generic method) — the same bug class already fixed for java/typescript/python/rust/csharp/kotlin/swift.
  • func_start/args never accepted backtick-quoted arbitrary identifiers at all — Scala's escape hatch for reserved-word/space-containing method names (e.g. Java-interop methods or ScalaTest-style spec names like def `should handle edge cases`(): Unit = {}). Fixed as an alternative capture group resolved via match.lastindex (existing infrastructure the pipeline already had for this shape, e.g. java's (init)|(constructor) groups), not a widened class. Kotlin shares this exact identifier-grammar gap, missed during Extraction hardening: kotlin #823's pass — filed as a separate follow-up (kotlin func_start/args missing backtick-identifier support (found during #825) #899) rather than reopening Extraction hardening: kotlin #823.
  • _dependency_capture had no statement-boundary logic at all: the previous flat [\w.{}\s,]+ class let \s (which matches newlines) bleed across a SECOND real import statement into the following unrelated line on multi-import files, so the second import was silently never detected. It also truncated Scala 3 wildcard imports (import scala.util.chaining.*) at the * and =>-renamed block imports (import scala.util.{Try => STry}) at the =>. Replaced with a properly bounded segmented-dotted-path grammar instead of just widening the character class further.

Differential Scan

crucible_check.py showed a real diff this time, not a zero diff — confirmed genuine, not a regression:

  • Verified against the real Kafka scala corpus: several files' detected dependency counts roughly doubled once the bleed-over bug was fixed (e.g. KafkaRaftManager.scala's direct-upstream count jumped 24 → 53, ReplicaManager.scala jumped 79 → 144).
  • The diff also rippled into 9 unrelated language/repo pairs (php/laravel_core, zig/zls, ruby/rails, yaml/ansible, rust/wasmtime, python/cython, assembly/hellosilicon, css/odoo, css/element) plus global ecosystem-summary aggregates. Confirmed this is expected, not a confinement violation: every one of those diff lines was a global/derived metric (Topological Coordinates, PageRank-derived blast-radius counts, corpus-relative percentiles, network-graph modularity/assortativity) — never a raw per-file signature count for a file in an untouched language. _dependency_capture feeds the cross-repo dependency DAG (network_risk_sensor.py's PageRank + spatial_mapper.py's 3D projection), which is computed over the whole scanned corpus as one graph, so this ripple is the expected behavior of fixing a real dependency-detection bug, not a bug of its own.
  • Golden master fixtures (tests/golden_master_audit.json, tests/golden_master_zero_dep_audit.json) re-blessed via crucible_check.py --update --yes; both modes now PASS cleanly.

Checked test_language_standards_strict.py for intentional scala behavior before finalizing (per c's #822 lesson) — no conflicts found; existing scala regression tests pass unmodified.

Also documents two known, unfixed limitations: func_start's Mode-B string-shielding gap reproduces via triple-quoted strings (now confirmed on a ninth language); _dependency_capture's unshielded-content_buffer gap reproduces via a commented-out import (same pipeline-wide issue already documented for every other language).

Test plan

  • pytest tests/extraction/languages/test_scala.py (63 tests) — run via bare pytest from an unrelated CWD to reproduce CI's invocation style
  • python -m pytest tests/ — 3995 passed, 1 deselected, 1 xfailed (pre-existing, unrelated)
  • python tests/tools/audit_check.py — clean (ruff/mypy/dead-key baselines unchanged)
  • python tests/tools/crucible_check.py — PASS on both venvs after re-blessing (see Differential Scan above)
  • Migrated scala's cases out of the four monolithic dict files into tests/extraction/languages/test_scala.py
  • Epic Epic: Harden the four extraction gauntlets (function/args/class/dependency) per language #813 and the methodology doc updated with new recurring bug classes 33-35
  • ReDoS scaling sweeps (n=2000→32000, linear) for all three regex changes

🤖 Generated with Claude Code

Hardens all four extraction gauntlets (func_start, args, class_start,
_dependency_capture) for scala per tests/extraction/how_to_harden_extraction.md.
A fresh sweep surfaced three real bugs:

- func_start/args shared the flat `\[[^\]]*\]` Rule-11 square-bracket
  generic-parameter step-over, breaking any nested generic bound
  (`def foo[T <: Comparable[T]](x: T): T = {`) -- the same bug class
  already fixed for java/typescript/python/rust/csharp/kotlin/swift.
- func_start/args never accepted backtick-quoted arbitrary identifiers
  (Scala's escape hatch for reserved-word/space-containing method names,
  e.g. Java-interop or ScalaTest-style spec names). Fixed as an
  alternative capture group resolved via match.lastindex, not a widened
  class. Kotlin shares this exact gap, missed during #823 -- filed as a
  separate follow-up.
- _dependency_capture had no statement-boundary logic at all: the flat
  `[\w.{}\s,]+` class let `\s` bleed across a second real import
  statement into the next unrelated line on multi-import files, so the
  second import was silently never detected. It also truncated Scala 3
  wildcard imports (`*`) and `=>`-renamed block imports. Replaced with a
  bounded segmented-dotted-path grammar.

crucible_check.py showed a real diff this time (not a zero diff):
confirmed genuine via the real Kafka scala corpus, where several files'
detected dependency counts roughly doubled once the bleed-over bug was
fixed (e.g. one file's direct-upstream count jumped 24 -> 53). The diff
also rippled into 9 unrelated language/repo pairs plus global ecosystem
aggregates -- confirmed as expected cross-repo dependency-graph/PageRank/
3D-projection ripple (every non-scala diff line was a global or derived
metric, never a raw per-file signature count for an untouched language),
not a confinement violation. Golden master fixtures re-blessed.

Checked test_language_standards_strict.py for intentional scala behavior
before finalizing -- no conflicts; existing scala regression tests pass
unmodified. Migrated scala's cases out of the four monolithic dict files
into tests/extraction/languages/test_scala.py (63 tests, up from ~15).
Two known limitations documented: func_start's Mode-B string-shielding
gap reproduces via triple-quoted strings (ninth language confirmed);
_dependency_capture's unshielded-content_buffer gap reproduces via a
commented-out import (same pipeline-wide issue as every other language).

Epic #813 and the methodology doc updated with recurring bug classes
33-35.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@squid-protocol
squid-protocol merged commit 4b884d1 into main Jul 31, 2026
28 checks passed
@squid-protocol
squid-protocol deleted the feature/825-extraction-hardening-scala branch July 31, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extraction hardening: scala

1 participant