Skip to content

repo-map's aggregate byte budget is poisoned by files it declines to parse #7247

Description

@JSONbored

Context

packages/loopover-engine/src/miner/repo-map.ts's buildRepoMap enforces two independent byte budgets while walking files: a per-file cap (maxSourceBytes) and an aggregate cap across all files (maxTotalSourceBytes). The loop (around line 258-282):

let totalSourceBytes = 0;
for (const [index, file] of files.entries()) {
  if (index >= maxFiles) { /* skip: resource_limit */ continue; }
  const languageName = resolveRepoMapLanguage(file.path);
  const sourceBytes = Buffer.byteLength(file.sourceText, "utf8");
  totalSourceBytes += sourceBytes;
  if (
    sourceBytes > maxSourceBytes ||
    totalSourceBytes > maxTotalSourceBytes
  ) {
    entries.push({ path: file.path, language: languageName, symbols: [], skipped: "resource_limit" });
    continue;
  }
  // ... file is actually parsed for symbols below
}

totalSourceBytes += sourceBytes runs unconditionally, before the check that decides whether this file gets skipped for being individually too large (sourceBytes > maxSourceBytes). So a single pathologically large file (e.g. a vendored/minified asset or generated bundle that slips into the files list the miner passes in) that is itself skipped for exceeding maxSourceBytes still consumes its full byte count against maxTotalSourceBytes — the aggregate budget meant to bound total parsed work is burned by a file that was never parsed at all. Every subsequent file in the same call, however small and legitimate, then also fails totalSourceBytes > maxTotalSourceBytes and gets marked skipped: "resource_limit", producing a near-empty repo map (most files unparsed) with no error surfaced to the coding-agent driver that consumes it — a silent, order-dependent, wrong-but-plausible result exactly matching this file's own resource-limiting intent gone wrong.

The existing test suite (repo-map.test.ts, per a prior gap-audit pass over this batch) only exercises the case where the first file is actually processed and legitimately consumes budget; it does not exercise a first file that is itself skipped yet still burns the aggregate budget for everyone after it.

Requirements

  • Only add a file's byte count to totalSourceBytes when that file is actually going to be parsed for symbols — i.e., after the individual sourceBytes > maxSourceBytes check has already determined the file is not being skipped for its own size, not before.
  • The aggregate maxTotalSourceBytes check must still work exactly as before for files that ARE within the per-file limit — this is a reordering fix only, not a change to either budget's semantics or default values.
  • A file skipped for being individually oversized must not affect whether any other file in the same call gets skipped for the aggregate budget.

Deliverables

  • Reorder the budget accounting in buildRepoMap so totalSourceBytes only accumulates bytes of files that pass the per-file maxSourceBytes check
  • Regression test: a files list whose first entry exceeds maxSourceBytes (and is correctly skipped as resource_limit) followed by several small files well within both budgets — assert the small files are NOT skipped and their symbols ARE parsed, proving the oversized first file no longer poisons the aggregate budget for the rest of the list

Test Coverage Requirements

This repo's Codecov patch gate is 99%+ hard (branch-counted) on every changed line/branch in src/**/packages/**. The regression test must reproduce the exact cascading-skip failure mode described (oversized-first-file → all subsequent small files wrongly skipped) and assert it's fixed.

Expected Outcome

A single oversized file early in a repo-map build no longer silently zeroes out the parsed-symbol coverage for every other file in the same call — the aggregate byte budget only counts bytes that were actually spent parsing.

Links & Resources

packages/loopover-engine/src/miner/repo-map.ts:219-220 (the two budget options), :258-282 (the loop with the ordering bug).

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions