feat(enrichment): full-file / near-verbatim duplication scan (#1520)

[dale053]

Summary

• Adds a DuplicationFinding type and duplication key to BriefFindings in review-enrichment/src/types.ts.
• Implements review-enrichment/src/analyzers/verbatim-duplication.ts: winnowing k-gram fingerprinting (k=8, w=4) over PR-added hunks vs the same-language repo tree at headSha. Two-phase: cheap whole-file containment gate (≥ 65%), then a sliding-window line scan to localize the source citation. Bounded at 150 files / 64 KB per file / 512 KB total / 8 concurrent blob fetches / 10 findings. Fail-safe on missing token, missing sha, or any network error.
• Registers the analyzer in src/brief.ts ANALYZERS registry with abort-signal forwarding.
• Renders a ### Near-verbatim code duplication block in src/render.ts, citing headFile:headLine duplicates sourceFile:sourceLine (~N lines, X% match) using safeCodeSpan for both paths.
• Adds 14 node:test unit and integration tests covering all exported functions, the render block, and the orchestrator wiring.

Closes #1520. Parent: #1499.

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked an issue, or this is small enough that the summary explains why an issue is not needed.

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally; global coverage stays at or above 97% for lines, statements, functions, and branches (aim for 98%+ branch coverage locally so CI variance does not fail near the threshold)
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

• The main gittensory CI suite (test:coverage, ui:*, build:mcp, etc.) is not applicable to this change: all code lives inside review-enrichment/, which is a standalone Node service with its own build and test pipeline (npm test inside review-enrichment/). All 58 tests pass (npm test in review-enrichment/), including the 14 new duplication-analyzer tests. The root-level CI checks listed above are for the Cloudflare Worker and do not cover review-enrichment/.

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
• API/OpenAPI/MCP behavior is updated and tested where needed.
• UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
• Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
• Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

No auth/CORS/session changes. No UI changes. The githubToken from EnrichRequest is used only in the Authorization header for GitHub API calls and is never logged or included in any finding output. The render block uses safeCodeSpan for all file paths (same pattern as every other analyzer).

UI Evidence

No UI changes.

Notes

• The review-enrichment/ service sits outside the main gittensory vitest/Codecov scope (per the issue: "outside the engine tsc/vitest/codecov scope"). Coverage is validated by the service's own node:test suite.
• Containment (fraction of block fingerprints found in source) is used instead of Jaccard because it correctly handles a short PR block being copied from a large source file — Jaccard would be low even for a perfect copy.
• The findBestSourceLine test uses a loose line-range assertion (>= 1 && <= 6) because multiple overlapping windows can legitimately score identically on the containment metric; the important invariant is that a match is found and the returned line is within the region containing the duplicated code.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[loopover-orb]

Caution

🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥

🛑 Gittensory review — closed

5 files · 1 AI reviewers · 2 blockers · readiness 55/100 · CI green · dirty

🛑 Closed — AI reviewers agree on a likely critical defect: review-enrichment/src/types.ts:103 leaves `DuplicationFinding` unterminated, so `CodeownersFinding` is parsed inside it and the file is syntactically invalid. — Resolve the flagged defect, or override if the AI reviewers are mistaken, then re-run the gate.; Readiness score is below the configured threshold — Address the short explicit PR panel actions, then re-run the gate.

Review summary
This change adds a duplication analyzer, wires it into the enrichment orchestrator, extends the findings contract, and renders duplication findings in the brief. The analyzer logic is bounded and fail-safe, but the submitted post-change files do not parse because two new blocks are left unterminated. Until those braces are restored, the package cannot build or run the new tests.

Blockers

• review-enrichment/src/types.ts:103 leaves `DuplicationFinding` unterminated, so `CodeownersFinding` is parsed inside it and the file is syntactically invalid.
• review-enrichment/src/render.ts:139 does not close the duplication `for` loop and `if` block before the CODEOWNERS section, so the renderer file is syntactically invalid and the CODEOWNERS rendering is accidentally nested in the duplication branch.

Why this is blocked

• AI reviewers agree on a likely critical defect: review-enrichment/src/types.ts:103 leaves `DuplicationFinding` unterminated, so `CodeownersFinding` is parsed inside it and the file is syntactically invalid.: review-enrichment/src/types.ts:103 leaves `DuplicationFinding` unterminated, so `CodeownersFinding` is parsed inside it and the file is syntactically invalid.
• Readiness score is below the configured threshold — Address the short explicit PR panel actions, then re-run the gate.

Signal	Result	Evidence
Code review	❌ 2 blockers	1 reviewers, synthesized
Linked issue	✅ Linked	#1520
Related work	⚠️ 3 scoped overlaps	Top overlaps are listed below; lower-confidence bulk is hidden.
Review load	❌ 8/20	Readiness component derived from cached public PR metadata and labels; size label size:L.
Validation evidence	❌ 5/25	Cached preflight status is hold.
Open PR queue	❌ 3/10	15 open PR(s), 10 likely reviewable, 5 unlinked.
Contributor context	✅ Confirmed Gittensor contributor	dale053; Gittensor profile; 39 PR(s), 21 issue(s).
Gate result	❌ Blocking	Repo-configured hard blocker found.

Nits — 6 non-blocking

• nit: review-enrichment/src/analyzers/verbatim-duplication.ts:260 uses `Buffer.from`, so confirm the enrichment runtime always has Node `Buffer` available or switch to a runtime-neutral base64 decoder.
• nit: review-enrichment/src/analyzers/verbatim-duplication.ts:225 silently treats a truncated recursive tree response as complete; GitHub can mark large recursive trees as truncated, so this should probably fail closed with no findings or be explicitly tested.
• nit: review-enrichment/test/enrichment.test.ts only asserts the duplication source line is positive, which leaves the localization behavior weakly covered despite being the main value of the second scan phase.
• Fix the missing `}` after `similarity: number;` in `review-enrichment/src/types.ts`.
• Close both the duplication `for` loop and `if` block before `const codeownersViolations` in `review-enrichment/src/render.ts`, then add a render test that includes both duplication and CODEOWNERS findings in one brief.
• AI maintainer-assist flagged possible low-effort patterns (elevated) — Advisory only — review the noted patterns; this AI assist never blocks the gate.

Review context

• Author: dale053
• Role context: outside_contributor
• Public audience mode: oss maintainer
• Lane context: Repository registration is not available in the local Gittensory cache.
• Public profile languages: not available
• Official Gittensor activity: 39 PR(s), 21 issue(s).
• Related work: Items reference the same linked issue feat(enrichment): Full-file / near-verbatim duplication scan #1520. (issue #1520)
• Related work: Open PR work references issue feat(enrichment): Full-file / near-verbatim duplication scan #1520. (issue #1520)
• Related work: Items reference the same linked issue MCP CLI: add PowerShell shell completion #1452. (issue #1452, PR #1448)
• Additional title-only matches omitted; title-only overlap does not block.

Contributor next steps

• Review top overlaps.
• Add scope summary.
• Fix blocker.
• Expect slower review.
• Refresh registry data or choose a registered active repo.
• Check active issues and PRs before submitting.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Review load = cached public PR metadata such as size labels, changed paths, and preflight status.
• Open PR queue = repo-wide review pressure; it is not a PR quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

Review details

Generated from public PR metadata and the diff. Advisory only; deterministic signals remain authoritative.

This change adds a duplication analyzer, wires it into the enrichment orchestrator, extends the findings contract, and renders duplication findings in the brief. The analyzer logic is bounded and fail-safe, but the submitted post-change files do not parse because two new blocks are left unterminated. Until those braces are restored, the package cannot build or run the new tests.

Blockers

• review-enrichment/src/types.ts:103 leaves `DuplicationFinding` unterminated, so `CodeownersFinding` is parsed inside it and the file is syntactically invalid.
• review-enrichment/src/render.ts:139 does not close the duplication `for` loop and `if` block before the CODEOWNERS section, so the renderer file is syntactically invalid and the CODEOWNERS rendering is accidentally nested in the duplication branch.

Nits (5)

• nit: review-enrichment/src/analyzers/verbatim-duplication.ts:260 uses `Buffer.from`, so confirm the enrichment runtime always has Node `Buffer` available or switch to a runtime-neutral base64 decoder.
• nit: review-enrichment/src/analyzers/verbatim-duplication.ts:225 silently treats a truncated recursive tree response as complete; GitHub can mark large recursive trees as truncated, so this should probably fail closed with no findings or be explicitly tested.
• nit: review-enrichment/test/enrichment.test.ts only asserts the duplication source line is positive, which leaves the localization behavior weakly covered despite being the main value of the second scan phase.
• Fix the missing `}` after `similarity: number;` in `review-enrichment/src/types.ts`.
• Close both the duplication `for` loop and `if` block before `const codeownersViolations` in `review-enrichment/src/render.ts`, then add a render test that includes both duplication and CODEOWNERS findings in one brief.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[JSONbored]

merge conflicts, please resubmit with all issues resolved. many of the analyzers issues touch similar files so the risk for churn does exist.