Story: BACKFILL + REMEDIATE — bring the fleet to green on the gitignore baseline #800

Activity

1. don-petry

   added

   enhancementFeature requestsFeature requests

   on Jul 19, 2026
2. don-petry commented on Jul 19, 2026

   

   don-petry

   on Jul 19, 2026

   ContributorAuthor

   More actions

   Part of Epic #801.
3. don-petry commented on Jul 19, 2026

   

   don-petry

   on Jul 19, 2026

   ContributorAuthor

   More actions

   ⚠️ Blocking finding from the fleet dry-run (2026-07-19) — do NOT apply as-is

   Ran scripts/sync-gitignore-baseline.sh --dry-run across the fleet. It plans 9 INSERT PRs (all non-exempt repos). Two problems make the real apply unsafe with the current upsert_gitignore_baseline (from #798):

   1. Negation regression (correctness — BLOCKER)

   upsert prepends the marker-wrapped L1 block and keeps the target's existing content below the END marker as "L2". For repos whose ad-hoc .gitignore has a broad ignore without the baseline's negation, that L2 line re-hides a file the L1 block just re-allowed. Verified with git check-ignore on the upsert output:

   • markets → public.pem becomes IGNORED (negation !public.pem defeated)
   • google-app-scripts → public.pem becomes IGNORED

   The baseline explicitly warns: "a later ignore silently re-hides a previously-negated file." The back-fill would ship exactly that.

   2. Full-baseline duplication (hygiene)

   Repos that already carry the old unmarkered baseline get the whole 398-line block duplicated:

   • TalkTerm 434 → 831 lines (.env, *.pem each appear twice)
   • same for bmad-bgreat-suite, ContentTwin

   Required before apply (added acceptance for #800)

   • upsert_gitignore_baseline must guarantee baseline negations win — e.g. strip/relocate any pre-existing L2 ignore that would re-hide a baseline-negated path (public.pem, .env.example, *.crt, ca.crt, *.pub, …), or place the block so its negations are evaluated last.
   • Detect an existing unmarkered baseline and migrate/dedup it (don't duplicate) — target repos with old baseline content converge to one marker-wrapped block + only their true L2.
   • Re-run --dry-run + the git check-ignore negation-safety probe on markets/google-app-scripts/TalkTerm → all re-allowed files stay un-ignored, no duplication.

   Evidence probe (reproducible): upsert each repo's live .gitignore, git init a temp dir with the result, git check-ignore .env.example .env.sample public.pem ca.crt.
4. don-petry

   added

   dev-leadFor dev-lead agent pickupFor dev-lead agent pickup

   on Jul 19, 2026
5. don-petry commented on Jul 19, 2026

   

   don-petry

   on Jul 19, 2026

   ContributorAuthor

   More actions

   Dev-Lead Implementation Plan

   Issue: #800 — Story: BACKFILL + REMEDIATE — bring the fleet to green on the gitignore baseline

   Scope

   Wire the shared, idempotent upsert_gitignore_baseline() (STORY2, scripts/lib/gitignore-baseline.sh) into the compliance remediation path so a missing/drifted push-protection/gitignore_baseline finding is auto-fixed via a PR (L2 preserved), instead of being routed to report_skip.

   Deliverable mapping

   • Deliverable 1 (Remediation hook) — implemented here as a code change in scripts/compliance-remediate.sh. This is the reviewable code artifact of this story.
   • Deliverables 2 & 3 (fleet back-fill PRs + admin-merge, live compliance-audit.sh re-run showing 0 findings) — these are operational runtime steps: they run the now-wired tooling against the 6 non-baseline repos and re-audit. They mutate other repos and cannot be part of this repo's code PR (see Constraints: "Do not commit, push, or open PRs"). They are executed by the compliance workflow/operator after this hook merges.

   Implementation Checklist

   • scripts/compliance-remediate.sh — source lib/gitignore-baseline.sh + lib/standards-deploy.sh; resolve SCRIPT_DIR/GITIGNORE_CANONICAL
   • scripts/compliance-remediate.sh — add remediate_gitignore_baseline() (extract canonical block → fetch repo .gitignore → upsert_gitignore_baseline → idempotent no-op skip → DRY_RUN → PR via sd_deploy_via_pr)
   • scripts/compliance-remediate.sh — route push-protection/gitignore_baseline to the new handler (remove the old report_skip)

   Tests to Write

   • test/scripts/compliance-remediate/gitignore-baseline.bats — new suite: INSERT (missing .gitignore) opens PR; REFRESH (drifted) opens PR; already-current is a no-op skip; DRY_RUN records planned PR with no mutating gh calls; L2 preserved in PUT payload; PR-create failure → report_fail
   • test/scripts/compliance-remediate/routing-messages.bats — update the gitignore_baseline case: it now lands on the remediation (PR) table, not skipped.md (behavior changed by this story)

   Test Command

   ./node_modules/.bin/bats --print-output-on-failure test/scripts/compliance-remediate/ test/scripts/lib/gitignore-baseline.bats && bash .dev-lead/scripts/dev-lead-lint.sh
6. don-petry commented on Jul 19, 2026

   

   don-petry

   on Jul 19, 2026

   ContributorAuthor

   More actions

   Dev-Lead: Implementation Complete

   Deliverable 1 — Remediation hook (this PR)

   Wired the shared, idempotent upsert_gitignore_baseline() into the compliance remediation path. push-protection/gitignore_baseline is no longer a manual report_skip — it now auto-fixes via a PR:

   • scripts/compliance-remediate.sh sources lib/gitignore-baseline.sh + lib/standards-deploy.sh.
   • New remediate_gitignore_baseline(): extracts the canonical L1 block from the org /.gitignore, fetches the target repo's .gitignore, runs upsert_gitignore_baseline() (L2 below END preserved verbatim), and ships the change through sd_deploy_via_pr() (PR only — never a direct push to a ruleset-protected default branch).
   • Idempotent: an already-current baseline is a no-op skip; a second run reuses the open PR (label gitignore-baseline, shared with sync-gitignore-baseline.sh) instead of duplicating it.
   • DRY_RUN=true records the planned PR and issues zero mutating gh calls.

   Plan Execution

   • Source gib + standards-deploy libs; resolve GITIGNORE_CANONICAL (test seam)
   • Add remediate_gitignore_baseline() (INSERT / REFRESH / no-op / dry-run / failure paths)
   • Route push-protection/gitignore_baseline to the new handler
   • New bats suite test/scripts/compliance-remediate/gitignore-baseline.bats (7 cases)
   • Update routing-messages.bats (finding is no longer a skip)
   • Gate the hook in CI (gitignore-baseline-tests.yml)

   Scope note — Deliverables 2 & 3 (operational, not code)

   Deliverable 2 (open back-fill PRs + admin-merge on the 6 non-baseline repos) and Deliverable 3 (re-run compliance-audit.sh fleet-wide to confirm 0 gitignore_baseline findings) are runtime operations that mutate other repos. They are executed by running this now-wired remediation (or sync-gitignore-baseline.sh) against the fleet after merge — they are not source changes and fall outside this repo's code PR (per the "do not commit/push/open PRs" constraint).

   Test Results

   1..17
   ok 1 opens a PR to INSERT the baseline when the repo has no .gitignore
   ok 2 opens a PR to REFRESH a drifted block and preserves L2 below END
   ok 3 marker-less .gitignore is preserved wholesale as L2 below the block
   ok 4 already-current baseline is an idempotent no-op (no PR opened)
   ok 5 dry-run records a planned PR and issues no mutating gh calls
   ok 6 failed PR creation is reported as a failure
   ok 7 an already-open baseline PR is reused (skip), not duplicated
   ok 8 secret_scan_ci_job_present skips with CI-job guidance, not apply-repo-settings
   ok 9 missing-copilot-instructions routes to the content-agent path
   ok 10 copilot-instructions-missing-tech-stack routes to the content-agent path
   ok 11 copilot-instructions-missing-local-dev-commands routes to the content-agent path
   ok 12 copilot-instructions-missing-language-links routes to the content-agent path
   ok 13 applies PATCH to security_and_analysis for non_provider_patterns_enabled
   ok 14 applies PATCH to security_and_analysis for secret_scanning_non_provider_patterns (legacy alias)
   ok 15 dry-run does not call the GitHub API but records the planned change
   ok 16 failed PATCH is reported as a failure
   ok 17 non-remediated push-protection findings still hit the skip path
   
   shellcheck -x scripts/compliance-remediate.sh → clean
   .dev-lead/scripts/dev-lead-lint.sh → all checks passed
   

   Files Changed

   • scripts/compliance-remediate.sh: source baseline+deploy libs; add remediate_gitignore_baseline(); route the finding to it (was a skip).
   • test/scripts/compliance-remediate/gitignore-baseline.bats: new suite — INSERT / REFRESH (L2 preserved) / marker-less / idempotent no-op / dry-run / PR-failure / reuse-open-PR.
   • test/scripts/compliance-remediate/routing-messages.bats: drop the obsolete "gitignore_baseline skips" case (behavior changed).
   • .github/workflows/gitignore-baseline-tests.yml: shellcheck + bats now cover compliance-remediate.sh and its remediate tests.

   Notes

   • Idempotency verified two ways: content-equality no-op, and label-keyed open-PR reuse (shared with the sync tool, so the two paths converge on one PR).
   • L2 preservation is enforced by the shared upsert_gitignore_baseline() and asserted end-to-end by decoding the PR's PUT payload in the new suite.
7. don-petry commented on Jul 19, 2026

   

   don-petry

   on Jul 19, 2026

   Last edited by don-petry

   ContributorAuthor

   More actions

   Blocking sub-task for the fleet apply filed as #809 (upsert negation-safety + dedup). #808 delivers the remediation hook (Part of #800); the apply stays gated until #809 merges and a clean re-dry-run + git check-ignore probe pass.
8. don-petry

   added 2 commits that reference this issue on Jul 19, 2026

   feat: implement issue #800 — Story: BACKFILL + REMEDIATE — bring the …

   ...

   2213266

   feat: implement issue #809 — upsert_gitignore_baseline must preserve …

   ...

   d1254f7
9. don-petry commented on Jul 19, 2026

   

   don-petry

   on Jul 19, 2026

   ContributorAuthor

   More actions

   Per go/no-go decision: refine output cleanliness before the fleet apply. Filed #817 (conditional negation tail + migrated-comment dedup) — blocks the apply. Tooling is validated & ready (dry-run clean, negation probe green on all 5 sampled repos); apply proceeds after #817 merges and a re-validation.
10. don-petry commented on Jul 20, 2026

    

    don-petry

    on Jul 20, 2026

    ContributorAuthor

    More actions

    ✅ Fleet back-fill complete — all 9 non-exempt repos now carry the marker-wrapped secrets baseline (single BEGIN/END block, negations verified safe, secrets ignored).

    Canary (verified on main): markets#343 (417 lines), TalkTerm#387 (433).
    Remaining 7 (merged): google-app-scripts#459, broodly#420, broodminder-export#54, ContentTwin#366, bmad-bgreat-suite#404, incubator#46, repo-template#69.

    repo-template#69 was admin-merged: its sole blocker was a pre-existing add-to-project infra failure (missing GitHub-App client-id secret in the repo — unrelated to this change), and one Gemini finding that was a false positive (it read the correct - removal of the dangling # Dev-lead agent working directory comment as 'left dangling'). Content verified clean before merge.

    Epic #801 delivered end-to-end: standard → enforcement (error severity) → seed/distribute → remediation hook → negation-safe upsert → clean fleet back-fill.
11. don-petry

    closed this as completedon Jul 20, 2026