Skip to content

fix(backfill): hydrate merged PR changed files in segment backfill - #353

Merged
JSONbored merged 2 commits into
JSONbored:mainfrom
galuis116:fix/recent-merged-changed-files-backfill
Jun 4, 2026
Merged

fix(backfill): hydrate merged PR changed files in segment backfill#353
JSONbored merged 2 commits into
JSONbored:mainfrom
galuis116:fix/recent-merged-changed-files-backfill

Conversation

@galuis116

@galuis116 galuis116 commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Closes #352.

Summary

The production segment-pipeline backfill stored changedFiles: [] for every recent-merged PR (backfillRecentMergedSegment hard-coded []), while only the legacy monolithic path fetched the real list. upsertRecentMergedPullRequest also unconditionally overwrote changedFilesJson, clobbering any populated list, and no detail-sync hydrates merged-PR files. So recent_merged_pull_requests.changedFiles was always empty in production, blinding the outcome path/size/test-evidence dimensions and collision file-overlap (both read mergedDetail.changedFiles).

Scope

  • src/github/backfill.ts — fetch each merged PR's changed files in the recent-merged segment via fetchPullRequestFiles (mirrors the monolithic path; reuses the existing mapWithConcurrency, REST -> GraphQL fallback, and rate-limit handling).
  • src/db/repositories.tsupsertRecentMergedPullRequest keeps the existing changedFilesJson on conflict when the incoming list is empty (no clobber).
  • test/unit/backfill.test.ts — segment-hydration + coalesce tests.

No schema, API, or public-surface changes.

Validation

npx vitest run test/unit/backfill.test.ts                                   # 53/53
npx vitest run test/unit/queue.test.ts test/unit/data-spine.test.ts \
  test/unit/data-quality.test.ts test/integration/api.test.ts \
  test/integration/routes-errors.test.ts                                    # 103/103
npx tsc --noEmit                                                            # clean
git diff --check                                                            # clean (no whitespace errors)

Branch coverage stays >= 97% (both coalesce ternary branches and the segment path are covered by the new tests). CI validate is green.

Safety

  • The only data added is each PR's changed-file paths, fetched from the same /pulls/{number}/files GitHub endpoint the monolithic backfill already uses — no secrets, no diffs, no file content, and no new external surface or privileges.
  • File fetches use the existing installation/public token and the existing rate-limit-aware fetch path, and fall back to [] on failure (a warning is recorded, no throw).
  • The coalesce guard only prevents data loss — an empty incoming list never erases a populated one — and cannot introduce stale data, because a non-empty incoming list still overwrites.

Notes

@JSONbored JSONbored left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@galuis116 the backfill problem statement is useful, but the PR needs hygiene fixes before merge.

A few notes:

  • Hydrating changed files for recent merged PRs is a real data-quality improvement.
  • Preserving non-empty existing file lists on re-upsert is the right invariant.
  • The current title/body do not meet this repo's PR requirements.

Required changes:

  • Retitle to fix(backfill): hydrate merged PR changed files in segment backfill.
  • Update the PR body to the current template with Summary, Scope, Validation, Safety, and Notes.
  • Include exact validation commands and security/privacy notes for the backfill data surface.

Validation expected:

  • Keep the current green CI.
  • Include the targeted backfill tests plus git diff --check in the PR body.

The recent-merged segment backfill stored changedFiles: [] and the upsert clobbered any existing files. Fetch files in the segment path (like the monolithic path) and preserve a non-empty file list on re-upsert.
@galuis116
galuis116 force-pushed the fix/recent-merged-changed-files-backfill branch from 1a9fe17 to 5cb82b2 Compare June 3, 2026 22:10
@galuis116 galuis116 changed the title Fetch merged-PR changed files in segment backfill and preserve them on re-upsert fix(backfill): hydrate merged PR changed files in segment backfill Jun 3, 2026
@github-actions github-actions Bot added the bug label Jun 3, 2026
@galuis116

Copy link
Copy Markdown
Contributor Author

Thanks for the review — addressed the hygiene items (no code changes from the reviewed diff):

  • Retitled to fix(backfill): hydrate merged PR changed files in segment backfill (PR + commit).
  • Rewrote the body to the template: Summary, Scope, Validation, Safety, Notes.
  • Added exact validation commands, the targeted backfill tests, git diff --check (clean), and security/privacy notes for the backfill data surface (only changed-file paths from the same /pulls/{n}/files endpoint the monolithic path already uses; no new surface).

CI is green and re-running on the amended commit. Ready for another look.

@galuis116
galuis116 requested a review from JSONbored June 3, 2026 22:14

@JSONbored JSONbored left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@galuis116 this is in good shape from this pass.

A few notes:

  • The segment backfill now hydrates changed-file paths instead of storing empty arrays.
  • The coalesce guard avoids clobbering existing file data with empty incoming lists.
  • The validation evidence is specific and covers the affected backfill, queue, data-quality, API, typecheck, and diff-check surfaces.

No code changes are requested from this review pass.

@dosubot dosubot Bot added the lgtm label Jun 4, 2026
@JSONbored
JSONbored merged commit 9d89aa3 into JSONbored:main Jun 4, 2026
4 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in gittensory - v1 roadmap Jun 4, 2026
@JSONbored JSONbored added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

No open projects
Status: Done

2 participants