Skip to content

Preserve plural Foundry test roots across invariant handoffs #212

Closed

Description

@aviggiano

Summary

The invariant-suite handoff assumes every repository uses test/, but Aave v4 at the benchmark commit 6959e3219b5506bf2acae18551cbb2a68a5b8fba uses the plural Foundry test root tests/ (tests/Base.t.sol, tests/invariants/...). The current generated workflow filters git ls-files to test/, requires test_paths to start with test/, and resets generated tests under test/foundry/. As a result, generated Recon/Chimera sources under a repository's existing tests/ convention can be omitted from the durable suite and downstream campaign worktrees.

Expected behavior

Detect and preserve the repository's actual test roots (at least test/ and tests/) while retaining path traversal/sensitive-root protections. Generated test materialization and retry cleanup should use the selected root, and implementation/test provenance should validate against the selected root(s). Add regression coverage using an Aave-like tests/ fixture.

Reproduction evidence

At the pinned Aave commit:

git ls-files test   # 0
git ls-files tests | head
# tests/Base.t.sol
# tests/Constants.sol
# tests/Utils.sol

This blocks the intended property specification -> invariant scaffolding -> Recon handoff from being reliable for the target.

Activity

  1. added 2 commits that reference this issue on Aug 7, 2026
  2. aviggiano commented on Aug 7, 2026

    @aviggiano
    CollaboratorAuthor

    Merge hazard — needs a human, not git.

    main implements plural roots as a list: INVARIANT_TEST_ROOT_NAMES (workflow.tsx:3326) and invariantTestRoots() (:3348-3358).

    The branch behind PR #165 (bench/deepseek-v4-flash-results) implements a different model for the same problem: it has no invariantTestRoots at all, and instead adds single-root invariantTestRootName() (:3610) plus repositoryAwareWorkspaceOutputRoots() (:3641).

    git merge-tree reports no conflict between them, so merging PR #165 will silently leave both root models in one file. Reconcile by hand before that merge.

    Also note PR #165 genuinely fixes this issue but carries no Closes #212, so merging it would close nothing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions