Skip to content

Make invariant-suite baseline provenance immutable across agent handoffs #213

Closed

Description

@aviggiano

Summary

The invariant-suite retry/handoff implementation stores invariant-suite-baseline.json under the task artifact directory, which is passed to the model agent as a writable addDir. After the agent returns, prepareArtifactMirror sees the existing sidecar and skips recapturing it; changedTestTreePaths then trusts the agent-controlled hashes. A model can therefore edit the sidecar to make generated or modified invariant files appear unchanged, preventing them from reaching the durable invariant-suite handoff.

Expected behavior

The baseline used for change detection must be immutable to the agent: derive changes from the pinned source ref, retain a process-owned digest, or store/verify a durable baseline outside agent-writable artifact roots. Retries and resumed runs must preserve the original baseline without recapturing after failed-attempt edits.

Reproduction shape

  1. Start an invariant stage and let preparation create invariant-suite-baseline.json.
  2. Modify a tests/... or test/... harness source and edit the baseline entry's SHA-256/size in the agent-visible artifact directory.
  3. Return from the agent.
  4. The post-agent materializer compares against the edited sidecar and omits the changed source from invariant-suite/.

The resulting downstream campaign can run without the generated CryticTester/Properties/TargetFunctions sources despite a successful implementation task.

Activity

  1. aviggiano commented on Aug 5, 2026

    @aviggiano
    CollaboratorAuthor

    Follow-up: an in-memory digest protects a single workflow process, but durable resume can restart the generated workflow with an empty map. Resume must reconstruct/verify the baseline against the pinned source (or another non-agent-mounted durable snapshot/signature), not accept an existing agent-writable sidecar when no in-memory digest exists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions