Skip to content

#3265 step 5: benchmark call-graph accuracy (tree-sitter call tags + type-aware tools) #3332

Closed

Description

@squid-protocol

Part of #3265. Depends on #3327 (contract) and #3328 (resolver). Required before any function-graph number is used in a README claim, a badge, or a headline score.

Two levels, scored separately:

Level 1: callee-name extraction. Precision/recall on (function, callee name) pairs per language, on the existing language-crucible corpus. Comparator: tree-sitter tagging queries (tags.scm @reference.call captures), which work at the same unresolved-name level. tree_sitter_language_pack is already in the audit venv. Reuse the tree-sitter accuracy audit's history CSV, chart and ledger pattern. ctags is not useful here: it indexes definitions, with little support for call references. Bonus: score the commit before #3267 against current main to get the missing before/after for Epic #3264.

Level 2: resolution correctness (the claim #3328 makes). On 3-4 flagship languages, compare resolved fcall edges with type-aware call graphs: PyCG or pyan3 (already in requirements.txt) for Python, and Sourcegraph SCIP indexers (scip-java, scip-typescript, scip-go) for the others. Report the wrong-target rate from name collisions. Mainframe: extend the hand-verified calls in tests/cobol_mainframe/answer_key/ to PL/I and JCL.

Control plants: add known-answer cases to keyword-rosetta:

  • calls inside strings and comments (must not count);
  • if ( / while ( look-alikes;
  • chained methods, constructors, functions passed by name.

Disagreements are verified against real source before they count (CLAUDE.md "comparative-correctness" rule).

Done when: a Level-1 CI audit is baselined for the corpus, and Level-2 numbers are published for at least Python plus one more language.

Metadata

Metadata

Assignees

No one assigned

    Labels

    core-engineModifications to the central physics and parsing engineenhancementNew feature, sensor, or structural signature

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions