You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
appmap-gold-traces/assets/manage.mjs never looks at a recording's metadata.test_status. update seeds or blesses whatever AppMap file the test run produced. If the run reaches update with a recording from a failed test, the failure becomes the gold baseline: the bug is recorded as the correct behavior, and the later fix shows up as a regression.
The only guard is runShell throwing on a non-zero exit from the record command, and that only applies to update --record.
How it happens
update without --record, on recordings produced by an earlier run that had failures.
update --record where the runner is configured not to fail the build on test failures (Gradle ignoreFailures = true, a -P flag some projects define for it, pytest ... || true, Maven -Dmaven.test.failure.ignore). Teams do this so a failing test still yields a recording to look at. The AppMap agents still write the file, with metadata.test_status: "failed".
Either way the exit code is 0 and update copies the file into gold_traces/baseline/appmaps/.
We hit this in the appmap-scan skill (evlawler/goldtrace-scan), where probes are expected to fail: the skill now works around it by never giving a failing probe a manifest entry until the fix passes. Any other gold-traces user can still hit it.
Proposed fix
In assessAppMap (or validateFreshEntries), treat a non-passing recording as unsuitable:
conststatus=appmap.metadata?.test_status;if(status&&status!=='succeeded'){errors.push(`test_status is "${status}"; a gold trace must come from a passing test`);}
Same check in recordCheckPass so check --record reports it too. Recordings with no test_status (non-test recordings) pass unchanged. The message should tell the user the recording is kept as evidence but not blessed.
Activity