Skip to content

test(graph): import-graph accuracy for 7 more languages (13 → 20) - #3594

Merged
squid-protocol merged 1 commit into
mainfrom
claude/festive-fermat-kojuy7
Sep 25, 2026
Merged

squid-protocol merged 1 commit into
mainfrom
claude/festive-fermat-kojuy7

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Summary

This extends tests/tools/import_graph_accuracy.py, the import graph measured against tree-sitter ground truth and gated in CI by graph-accuracy-audit.yml, from 13 to 20 languages. Each new language gets a pinned real repo and a resolver that applies the language's own import rule:

language repo rule
objective-c SDWebImage the C rule (#import = #include once); @import Module; is external
dart dart-lang/http package:p/x.dart → <p's pubspec dir>/lib/x.dart; dart: is the SDK; other URIs are relative. Covers import, export, part and part of.
kotlin kotlinpoet by package + declared name, from an index of every file's package and top-level declarations (neither language ties path or file name to content). Member or nested imports fall back to the owner's file; a Java class is found by Java's rule.
scala circe as Kotlin, plus {A, B => C} selectors, _/* wildcards, package objects, and names relative to each enclosing package. Backquoted segments (io.circe.`export`) are normalized.
haskell shellcheck import A.B.C → a file ending in A/B/C.hs (.lhs, .hsc)
shell bash-it source p / . p: relative to the file, else by suffix. A path behind a variable keeps its literal tail.
solidity openzeppelin-contracts relative imports exact; @scope/pkg/x.sol through a package.json named @scope/pkg, else from the project root; otherwise a dependency.

COBOL is deliberately not added. tree-sitter-cobol has no EXEC SQL / EXEC CICS rule, so its error recovery swallows the COPY statements that follow one. In CBSA's ACCTCTRL.cbl, one ERROR node spans lines 85–127 and takes three COPYs with it. It found 64 of CBSA's ~148 COPY/INCLUDE statements. COBOL's copybook and call edges are already scored exactly against the hand-verified answer keys (ground_truth_ledger.py, every PR): copybooks 422/422 and program calls 142/142 across the six corpora. The docstring points there.

Baseline (precision / recall)

language precision recall notes
haskell 100 100
objective-c 99.8 97.3 Misses: SDWebImage ships each public header twice (Core/ and include/SDWebImage/), so a bare #import "X.h" from Private/ is ambiguous and draws nothing (#261).
solidity 97.1 99.8 24 "false positives" import ../patched/..., a directory the formal-verification Makefile generates. The engine links the unpatched contracts/ original by name, which is arguably right.
dart 99.1 80.8 Engine gaps: no rule for package: URIs; imports in /// doc comments are captured.
kotlin 100 62.1 Engine gap: top-level functions and member imports (a.b.Util.X) live in files not named after them.
shell 87.5 75.0 Engine gap: variable-led paths ("${BASH_IT?}/themes/…", "$rvm_path/scripts/completion") resolve by bare stem.
scala 99.4 33.0 Engine bug: import a.b.{ C, D } is split before the prefix is distributed, so the tokens are a.b. C and D.

The existing 13 languages are unchanged. The engine findings get their own issues; this PR only adds the measurement.

Two ground-truth errors were found and fixed while validating disagreements against source, per CLAUDE.md's comparative-correctness rule:

  • A Kotlin file importing a Java class.
  • A backquoted Scala package segment.

In both cases the engine was right.

Type of change

  • Bug fix
  • New feature / language support
  • Parsing or engine logic
  • Docs, tooling, or CI only
  • Other (describe above)

CI checklist

  • ruff-audit: no new findings. Changed files formatted.
  • pytest tests/tools/test_import_graph_accuracy.py: 14 passed. Plain pytest from another directory also works; the tree-sitter tests skip there.
  • import_graph_accuracy.py --regenerate: all 20 languages measured, baseline rewritten.

Verification

$ python tests/tools/import_graph_accuracy.py --fetch-only    # 7 new pinned repos
$ python tests/tools/import_graph_accuracy.py --regenerate
... 20 languages; existing 13 identical to the previous baseline
$ pytest tests/tools/test_import_graph_accuracy.py -q    -- 14 passed

🤖 Generated with Claude Code

https://claude.ai/code/session_01YAtpBfGGyBGGovitSVaM5Q


Generated by Claude Code

Adds tree-sitter ground-truth resolvers and a pinned real repo for each:
objective-c (SDWebImage), dart (dart-lang/http), kotlin (kotlinpoet),
scala (circe), haskell (shellcheck), shell (bash-it), solidity
(openzeppelin-contracts). Each resolves an import by the language's own rule
-- Kotlin/Scala by package + declared name from an index of every file, Dart
through pubspec.yaml package names, Solidity through package.json names.

COBOL is deliberately not added: tree-sitter-cobol has no EXEC SQL/CICS rule
and its error recovery swallows the COPYs after one (64 of CBSA's ~148). Its
copybook and call edges are already scored exactly against the hand-verified
answer keys (ground_truth_ledger.py); the docstring says so.

Baseline (precision / recall): haskell 100/100, objective-c 99.8/97.3,
solidity 97.1/99.8, dart 99.1/80.8, kotlin 100/62.1, shell 87.5/75.0,
scala 99.4/33.0. The existing 13 are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YAtpBfGGyBGGovitSVaM5Q
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

@squid-protocol
squid-protocol merged commit 981769b into main Sep 25, 2026
30 checks passed
@squid-protocol
squid-protocol deleted the claude/festive-fermat-kojuy7 branch September 25, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants