Skip to content

feat(miner-hands): tree-sitter-based repo map builder - #4542

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
oktofeesh1:feat/miner-repo-map-4280
Jul 10, 2026
Merged

feat(miner-hands): tree-sitter-based repo map builder#4542
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
oktofeesh1:feat/miner-repo-map-4280

Conversation

@oktofeesh1

Copy link
Copy Markdown
Contributor

Summary

A repo-map builder gives a coding-agent driver (or the acceptance-criteria/prompt-packet builders upstream of it) a compact, structural view of a target repository — function/class/method/interface/type signatures — without paying the token cost of dumping every file's full contents into the prompt.

  • buildRepoMap(files, options?): parses each file with web-tree-sitter (the WASM binding, not a native addon — this package also ships a Cloudflare Workers deployment target where native Node addons are not an option) using prebuilt grammars from tree-sitter-wasms, and extracts a RepoMapFileEntry per file.
  • Supported today: JavaScript/TypeScript/TSX (this repo's own dominant languages). Also handles anonymous export default class {} / export default function() {} expressions (reported with an "<anonymous>" name) rather than silently dropping them.
  • Graceful degradation: a file with an unsupported extension, or a grammar that fails to load, or a parse that throws, is reported via skipped: "unsupported_language" | "grammar_unavailable" on its entry — never thrown. A language's grammar is loaded (and cached) at most once per buildRepoMap call even across many files.
  • renderRepoMap(entries, maxOutputChars?): a pure renderer producing a bounded plain-text outline, truncating (with a marker line) once the char budget would be exceeded, so it can't blow out a prompt budget on a large repo.
  • Known scope limit (documented in the module header): arrow functions and expressions bound via a const foo = ... declarator are not walked up to their binding identifier — an anonymous class/function expression reports "<anonymous>" rather than resolving a variable name. Reasonable follow-up, not attempted here.
  • Adds web-tree-sitter and tree-sitter-wasms as new gittensory-engine dependencies — npm audit reports 0 findings. Verified npm run ui:build still tree-shakes them out of the Cloudflare Workers bundle (src/ never imports the miner/ subpath of the engine).

Closes #4280

Test plan

  • test/unit/repo-map.test.ts (25 tests, 100% statement/branch/function/line coverage on the new file): extension→grammar mapping, real parses of JS/TS/TSX fixtures (function/class/method/interface/type-alias extraction, 1-indexed line numbers, anonymous class/function expressions), signature truncation, unsupported-extension skip (without attempting any grammar load), grammar_unavailable on both a rejecting loader and a loader returning a language that fails to parse, per-language grammar-load caching across multiple files, and renderRepoMap's bounded output (normal render, skipped/empty-file placeholders, multi-symbol files, and truncation triggered at every distinct line-kind: header, skipped, no-symbols, and mid-symbol-list)
  • Manually ran buildRepoMap + renderRepoMap against a real file in this repo (packages/gittensory-engine/src/miner/attempt-metering.ts) and confirmed the extracted outline matches the file's actual types/functions
  • npm run typecheck
  • npm audit --audit-level=moderate (0 vulnerabilities)
  • npm run ui:build (Workers bundle unaffected by the new dependencies)
  • npm run test:ci (full local gate, green)

Add buildRepoMap/renderRepoMap: a compact, structural view of a
target repository (function/class/method/interface/type signatures)
for a coding-agent driver or the acceptance-criteria/prompt-packet
builders upstream of it, without paying the token cost of dumping
full file contents into a prompt.

Parses with web-tree-sitter (the WASM binding, not a native addon --
this package also ships a Cloudflare Workers deployment target where
native Node addons are not an option) using prebuilt grammars from
tree-sitter-wasms. Supports JavaScript/TypeScript/TSX today.

Graceful degradation: an unsupported file extension or a grammar that
fails to load/parse is reported via `skipped` on its RepoMapFileEntry,
never thrown -- this module extracts what it safely can rather than
blocking the whole driver invocation. renderRepoMap bounds its output
to a configurable char budget, truncating (with a marker) rather than
blowing out a prompt budget on a large repo.

Adds web-tree-sitter and tree-sitter-wasms as new gittensory-engine
dependencies (0 npm audit findings); verified npm run ui:build still
tree-shakes them out of the Cloudflare Workers bundle since src/ never
imports the miner/ subpath.

Closes JSONbored#4280
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.05%. Comparing base (95e9ec6) to head (b26652c).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4542      +/-   ##
==========================================
+ Coverage   94.04%   94.05%   +0.01%     
==========================================
  Files         422      423       +1     
  Lines       37600    37672      +72     
  Branches    13736    13756      +20     
==========================================
+ Hits        35362    35434      +72     
  Misses       1583     1583              
  Partials      655      655              
Files with missing lines Coverage Δ
packages/gittensory-engine/src/miner/repo-map.ts 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. gittensor:priority Maintainer-selected Gittensor priority — scores a 1.5x multiplier. labels Jul 10, 2026
@loopover-orb

loopover-orb Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

Review updated: 2026-07-10 00:36:36 UTC

6 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This adds a well-scoped, self-contained tree-sitter-based repo-map builder (`buildRepoMap`/`renderRepoMap`) plus barrel exports, using the WASM binding (not a native addon) so it stays deployable from the same package that ships to Cloudflare Workers. The implementation is correct on inspection: language resolution, per-language grammar caching, graceful `skipped` degradation (never throwing), and the char-budget-bounded renderer all do what they claim, and the tests exercise real code paths (e.g. the `grammar_unavailable` tests inject a loader/language that make the real `try/catch` in `buildRepoMap` fire, rather than fabricating an unreachable branch). CI is green and `npm audit` is clean per the description; nothing here breaks a reachable path.

Nits — 5 non-blocking
  • packages/gittensory-engine/src/miner/repo-map.ts:150-176 (`buildRepoMap`'s per-file loop) nests to depth 5 (extension check → language resolve check → try/catch → push) — consider extracting the per-file body into a small `buildEntryForFile` helper for readability.
  • packages/gittensory-engine/src/miner/repo-map.ts:110/141/188 — `120` and `20_000` are reasonable defaults but are unnamed magic numbers in two different places; a `DEFAULT_MAX_SIGNATURE_CHARS`/`DEFAULT_MAX_OUTPUT_CHARS` constant would make the intent (and the relationship between the two call sites) explicit.
  • package-lock.json's new `tree-sitter-wasms` entry lists itself as a dependency (`"dependencies": { "tree-sitter-wasms": "^0.1.11" }`) — worth a sanity check that this isn't a corrupted/hand-edited lockfile entry rather than genuine upstream metadata.
  • buildRepoMap awaits files sequentially in a `for` loop (repo-map.ts:148) — fine at today's scale, but if this is meant to run over a full repo's file set consider `Promise.all` per resolved language now that grammar loads are already deduplicated by name.
  • The PR description references issue feat(miner-hands): tree-sitter-based repo map builder #4280 throughout the code/comments but doesn't state it closes/links a maintainer-authorized open issue — worth confirming that link explicitly per this repo's issue-scope convention.
Signal Result Evidence
Code review ✅ No blockers 1 reviewer
Linked issue ✅ Linked #4280
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 34 registered-repo PR(s), 27 merged, 0 issue(s).
Contributor context ✅ Confirmed Gittensor contributor oktofeesh1; Gittensor profile; 34 PR(s), 0 issue(s).
Gate result ✅ Passing No configured blocker found.
Linked issue satisfaction

Addressed
The PR adds web-tree-sitter/tree-sitter-wasms as specified, implements buildRepoMap/renderRepoMap with bounded output, graceful skip-not-throw handling for unsupported languages and grammar/parse failures, and a substantial test suite plus README documentation of supported languages and the module's design/limitations, covering essentially all deliverables.

Review context
  • Author: oktofeesh1
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 34 PR(s), 0 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gittensory approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit 604d971 into JSONbored:main Jul 10, 2026
10 checks passed
@loopover-orb loopover-orb Bot added gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. and removed gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. gittensor:priority Maintainer-selected Gittensor priority — scores a 1.5x multiplier. labels Jul 10, 2026
@JSONbored JSONbored added gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. gittensor:priority Maintainer-selected Gittensor priority — scores a 1.5x multiplier. and removed gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. labels Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. gittensor:priority Maintainer-selected Gittensor priority — scores a 1.5x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(miner-hands): tree-sitter-based repo map builder

2 participants