Repository navigation
test(graph): import-graph accuracy for 7 more languages (13 → 20) - #3594
Merged
Merged
Conversation
Adds tree-sitter ground-truth resolvers and a pinned real repo for each: objective-c (SDWebImage), dart (dart-lang/http), kotlin (kotlinpoet), scala (circe), haskell (shellcheck), shell (bash-it), solidity (openzeppelin-contracts). Each resolves an import by the language's own rule -- Kotlin/Scala by package + declared name from an index of every file, Dart through pubspec.yaml package names, Solidity through package.json names. COBOL is deliberately not added: tree-sitter-cobol has no EXEC SQL/CICS rule and its error recovery swallows the COPYs after one (64 of CBSA's ~148). Its copybook and call edges are already scored exactly against the hand-verified answer keys (ground_truth_ledger.py); the docstring says so. Baseline (precision / recall): haskell 100/100, objective-c 99.8/97.3, solidity 97.1/99.8, dart 99.1/80.8, kotlin 100/62.1, shell 87.5/75.0, scala 99.4/33.0. The existing 13 are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YAtpBfGGyBGGovitSVaM5Q
This was referenced Sep 25, 2026
Closed
Contributor
6 of 9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This extends
tests/tools/import_graph_accuracy.py, the import graph measured against tree-sitter ground truth and gated in CI bygraph-accuracy-audit.yml, from 13 to 20 languages. Each new language gets a pinned real repo and a resolver that applies the language's own import rule:#import=#includeonce);@import Module;is externalpackage:p/x.dart→<p's pubspec dir>/lib/x.dart;dart:is the SDK; other URIs are relative. Covers import, export, part and part of.packageand top-level declarations (neither language ties path or file name to content). Member or nested imports fall back to the owner's file; a Java class is found by Java's rule.{A, B => C}selectors,_/*wildcards, package objects, and names relative to each enclosing package. Backquoted segments (io.circe.`export`) are normalized.import A.B.C→ a file ending inA/B/C.hs(.lhs,.hsc)source p/. p: relative to the file, else by suffix. A path behind a variable keeps its literal tail.@scope/pkg/x.solthrough apackage.jsonnamed@scope/pkg, else from the project root; otherwise a dependency.COBOL is deliberately not added. tree-sitter-cobol has no
EXEC SQL/EXEC CICSrule, so its error recovery swallows theCOPYstatements that follow one. In CBSA'sACCTCTRL.cbl, one ERROR node spans lines 85–127 and takes three COPYs with it. It found 64 of CBSA's ~148 COPY/INCLUDE statements. COBOL's copybook and call edges are already scored exactly against the hand-verified answer keys (ground_truth_ledger.py, every PR): copybooks 422/422 and program calls 142/142 across the six corpora. The docstring points there.Baseline (precision / recall)
Core/andinclude/SDWebImage/), so a bare#import "X.h"fromPrivate/is ambiguous and draws nothing (#261).../patched/..., a directory the formal-verification Makefile generates. The engine links the unpatchedcontracts/original by name, which is arguably right.package:URIs; imports in///doc comments are captured.a.b.Util.X) live in files not named after them."${BASH_IT?}/themes/…","$rvm_path/scripts/completion") resolve by bare stem.import a.b.{ C, D }is split before the prefix is distributed, so the tokens area.b. CandD.The existing 13 languages are unchanged. The engine findings get their own issues; this PR only adds the measurement.
Two ground-truth errors were found and fixed while validating disagreements against source, per CLAUDE.md's comparative-correctness rule:
In both cases the engine was right.
Type of change
CI checklist
pytest tests/tools/test_import_graph_accuracy.py: 14 passed. Plainpytestfrom another directory also works; the tree-sitter tests skip there.import_graph_accuracy.py --regenerate: all 20 languages measured, baseline rewritten.Verification
🤖 Generated with Claude Code
https://claude.ai/code/session_01YAtpBfGGyBGGovitSVaM5Q
Generated by Claude Code