Description
Follow-up to #17 and #18. Today a review's findings live only in the PR comment: they have no identity, no state, and no memory across runs. A reviewer who has decided a 🟡 is "pre-prod configuration, not a bug" sees it again on the next push; a developer who has fixed a 🔴 in a separate branch has nowhere to say so; and nothing links the finding to the code that fixes it.
Proposal: a findings manifest the skill reads and writes
The skill records every 🔴/🟡 (and optionally 🟢) finding to a manifest committed next to the gold traces, e.g. gold_traces/findings.yml, one entry per finding with a stable id:
- id: openid-login-clears-admin-flag # slug derived from the ticket title; stable across runs
severity: high # red | medium | low
introduced_by: head # head | latent (present on base)
title: OpenID login must not clear an existing user's administrator flag when openid.adminGroup is unset
where:
- src/main/java/org/traccar/database/OpenIdProvider.java#handleCallback
- src/main/java/org/traccar/api/security/LoginService.java#login
expected: With no openid.adminGroup configured the stored administrator flag is left untouched on login.
actual: handleCallback yields administrator=false when the group is unset; LoginService.login writes it back.
evidence:
recording: OpenIdLoginProbeTest.withoutAnAdminGroupAnExistingAdministratorStaysAnAdministrator
base: reads tc_users, no write
head: DatabaseStorage#updateObject sets administrator=false
done_when: the probe recording on the fixed revision shows no UPDATE of tc_users.administrator; the configured-group recording still shows the sync
first_seen: 2026-09-23 head: 3f1c2a9
status: open # open | dismissed | linked | fixedThe user resolves an entry by editing status (by hand or via a appmap-review resolve <id> helper):
| status | fields | meaning |
|---|---|---|
dismissed |
reason, by, date |
not a defect here — e.g. "pre-prod env only: adminGroup is always set in prod config". The skill stops reporting it, but lists it once under "Dismissed" with the reason so the decision is visible on the PR. |
linked |
fix: <draft PR / branch / commit URL> |
a fix exists elsewhere. On the next run the skill records that revision's traces for the evidence test and checks done_when; if satisfied it sets status: fixed and says so in the comment, otherwise it reports "linked fix does not yet satisfy done_when". |
fixed |
fixed_in: <commit> |
set by the skill, never by hand. |
What changes in the PR comment
- Findings carry their id; re-runs say new / still open / dismissed (reason) / linked → PR#… / fixed in … instead of re-asserting everything.
- A finding is never silently dropped: if the code that triggered it disappears, the skill marks it
fixedwith the commit. - Optional: for each open finding the skill may attach a proposed fix as a suggested patch in the comment (it already knows where/expected/actual/done_when). Keep it as a suggestion or a draft PR link — never auto-apply.
Why in the repo, not the comment
The manifest is diffable, survives PR churn, works with any host (GitHub/GitLab/none), and gives an AI developer a single file to work from: pick an open entry, implement until done_when holds, set linked. The playground (evlawler/appmap-playground) is a good place to prototype the rendering: the Review tab would show each finding's status and the dismiss/link controls writing to the manifest.
Activity