Skip to content

appmap-gold-traces: manage.mjs update blesses recordings from failed tests (never checks metadata.test_status) #20

Open

Description

@devin-ai-integration

Summary

appmap-gold-traces/assets/manage.mjs never looks at a recording's metadata.test_status. update seeds or blesses whatever AppMap file the test run produced. If the run reaches update with a recording from a failed test, the failure becomes the gold baseline: the bug is recorded as the correct behavior, and the later fix shows up as a regression.

Where

  • updateBaseline → validateFreshEntries → assessAppMap: checks event count, size, repetition, project coverage. Nothing reads appmap.metadata.test_status. (grep test_status manage.mjs returns nothing.)
  • The only guard is runShell throwing on a non-zero exit from the record command, and that only applies to update --record.

How it happens

  1. update without --record, on recordings produced by an earlier run that had failures.
  2. update --record where the runner is configured not to fail the build on test failures (Gradle ignoreFailures = true, a -P flag some projects define for it, pytest ... || true, Maven -Dmaven.test.failure.ignore). Teams do this so a failing test still yields a recording to look at. The AppMap agents still write the file, with metadata.test_status: "failed".

Either way the exit code is 0 and update copies the file into gold_traces/baseline/appmaps/.

We hit this in the appmap-scan skill (evlawler/goldtrace-scan), where probes are expected to fail: the skill now works around it by never giving a failing probe a manifest entry until the fix passes. Any other gold-traces user can still hit it.

Proposed fix

In assessAppMap (or validateFreshEntries), treat a non-passing recording as unsuitable:

const status = appmap.metadata?.test_status;
if (status && status !== 'succeeded') {
  errors.push(`test_status is "${status}"; a gold trace must come from a passing test`);
}

Same check in recordCheckPass so check --record reports it too. Recordings with no test_status (non-test recordings) pass unchanged. The message should tell the user the recording is kept as evidence but not blessed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions