Skip to content

forge test --mutate: all mutants silently reported as INVALID (0% score) for certain --match-contract selections; underlying compile error is swallowed #15101

Description

@grandizzy

Component

Forge (forge test --mutate)

Describe the bug

When the test selection passed to mutation testing includes certain test
contracts, every mutant is classified as INVALID ("mutation produced a
compilation error") and the run reports a meaningless Mutation Score: 0.0%
(0/0)
— even though the exact same mutations are valid and killable with a
slightly different selection, and the project compiles and tests green normally.

The per-mutant compile is scoped to the file set from
TestArgs::get_sources_to_compile() (with dynamic_test_linking force-enabled
for mutation). For some selected test contracts this scoped compile returns
Err, and crates/forge/src/mutation/runner.rs maps that Err directly to
MutationResult::Invalid without surfacing the compiler error. The result
is a silent, misleading 0% score with no way to tell a tooling/compile failure
apart from genuinely non-compiling mutants.

To Reproduce

git clone --recurse-submodules https://github.com/morpho-org/morpho-blue
cd morpho-blue

# BROKEN: includes MorphoStorageLibTest -> 92/92 INVALID, score 0.0%
FOUNDRY_PROFILE=no_via_ir FOUNDRY_OPTIMIZER_RUNS=200 \
  forge test --mutate src/libraries/SharesMathLib.sol \
  --mc "SupplyIntegrationTest|WithdrawIntegrationTest|MorphoStorageLibTest" \
  --mutation-timeout 50

# WORKS: drop MorphoStorageLibTest -> mutants killed/survived correctly (~95%)
FOUNDRY_PROFILE=no_via_ir FOUNDRY_OPTIMIZER_RUNS=200 \
  forge test --mutate src/libraries/SharesMathLib.sol \
  --mc "SupplyIntegrationTest|WithdrawIntegrationTest" \
  --mutation-timeout 50

The selected contracts run fine as a normal test (no mutation):

FOUNDRY_PROFILE=no_via_ir forge test \
  --mc "SupplyIntegrationTest|WithdrawIntegrationTest|MorphoStorageLibTest"
# -> all green

Other selections also trigger it (e.g. any allowlist including
MorphoStorageLibTest, and at least one of the
Authorization/Callbacks/OnlyOwner integration tests); --no-match-contract
denylists that leave those contracts in the set hit it too. MathLibTest,
MarketParamsLibTest, ExtSloadIntegrationTest, etc. do not trigger it.

Broken output

╭──────────┬───────────┬────────────╮
│ Status   ┆ # Mutants ┆ % of Total │
╞══════════╪═══════════╪════════════╡
│ Survived ┆ 0         ┆ 0.0%       │
│ Killed   ┆ 0         ┆ 0.0%       │
│ Invalid  ┆ 92        ┆ 100.0%     │
│ Skipped  ┆ 0         ┆ 0.0%       │
╰──────────┴───────────┴────────────╯
Mutation Score: 0.0% (0/0 mutants killed)

Expected behavior

  • Mutants should be classified identically regardless of which (compiling,
    passing) test contracts are selected; valid mutations must not be reported as
    INVALID.
  • If a per-mutant compile genuinely fails, the compiler error should be surfaced
    (at least under -v/-vvv), so an infra/tooling compile failure is
    distinguishable from a non-typechecking mutation.
  • A run that produces 0/0 real results should not be presented as a 0.0%
    score; it should warn that no mutant was actually evaluated.

Environment

  • forge 1.7.2-dev (commit eba698dcf, master @ the native-symbolic merge)
  • macOS (arm64), solc 0.8.19

Additional context

Relevant code: TestArgs::get_sources_to_compile() in
crates/forge/src/cmd/test/mod.rs and the Err(_) => MutationResult::Invalid
mapping in crates/forge/src/mutation/runner.rs (compile_and_test).
config.dynamic_test_linking is force-enabled for mutation runs, which may be
implicated in the scoped-compile failure.

Metadata

Metadata

Assignees

Labels

Type

No type
No fields configured for issues without a type.

Projects

Status
Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions