Skip to content

appmap-review: record findings to a manifest with dismiss / link-to-fix / fixed states across runs #19

Open

Description

@devin-ai-integration

Follow-up to #17 and #18. Today a review's findings live only in the PR comment: they have no identity, no state, and no memory across runs. A reviewer who has decided a 🟡 is "pre-prod configuration, not a bug" sees it again on the next push; a developer who has fixed a 🔴 in a separate branch has nowhere to say so; and nothing links the finding to the code that fixes it.

Proposal: a findings manifest the skill reads and writes

The skill records every 🔴/🟡 (and optionally 🟢) finding to a manifest committed next to the gold traces, e.g. gold_traces/findings.yml, one entry per finding with a stable id:

- id: openid-login-clears-admin-flag            # slug derived from the ticket title; stable across runs
  severity: high                                # red | medium | low
  introduced_by: head                           # head | latent (present on base)
  title: OpenID login must not clear an existing user's administrator flag when openid.adminGroup is unset
  where:
    - src/main/java/org/traccar/database/OpenIdProvider.java#handleCallback
    - src/main/java/org/traccar/api/security/LoginService.java#login
  expected: With no openid.adminGroup configured the stored administrator flag is left untouched on login.
  actual: handleCallback yields administrator=false when the group is unset; LoginService.login writes it back.
  evidence:
    recording: OpenIdLoginProbeTest.withoutAnAdminGroupAnExistingAdministratorStaysAnAdministrator
    base: reads tc_users, no write
    head: DatabaseStorage#updateObject sets administrator=false
  done_when: the probe recording on the fixed revision shows no UPDATE of tc_users.administrator; the configured-group recording still shows the sync
  first_seen: 2026-09-23  head: 3f1c2a9
  status: open                                  # open | dismissed | linked | fixed

The user resolves an entry by editing status (by hand or via a appmap-review resolve <id> helper):

status fields meaning
dismissed reason, by, date not a defect here — e.g. "pre-prod env only: adminGroup is always set in prod config". The skill stops reporting it, but lists it once under "Dismissed" with the reason so the decision is visible on the PR.
linked fix: <draft PR / branch / commit URL> a fix exists elsewhere. On the next run the skill records that revision's traces for the evidence test and checks done_when; if satisfied it sets status: fixed and says so in the comment, otherwise it reports "linked fix does not yet satisfy done_when".
fixed fixed_in: <commit> set by the skill, never by hand.

What changes in the PR comment

  • Findings carry their id; re-runs say new / still open / dismissed (reason) / linked → PR#… / fixed in … instead of re-asserting everything.
  • A finding is never silently dropped: if the code that triggered it disappears, the skill marks it fixed with the commit.
  • Optional: for each open finding the skill may attach a proposed fix as a suggested patch in the comment (it already knows where/expected/actual/done_when). Keep it as a suggestion or a draft PR link — never auto-apply.

Why in the repo, not the comment

The manifest is diffable, survives PR churn, works with any host (GitHub/GitLab/none), and gives an AI developer a single file to work from: pick an open entry, implement until done_when holds, set linked. The playground (evlawler/appmap-playground) is a good place to prototype the rendering: the Review tab would show each finding's status and the dismiss/link controls writing to the manifest.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions