Component
Forge (forge test --mutate)
Describe the bug
When the test selection passed to mutation testing includes certain test
contracts, every mutant is classified as INVALID ("mutation produced a
compilation error") and the run reports a meaningless Mutation Score: 0.0%
(0/0) — even though the exact same mutations are valid and killable with a
slightly different selection, and the project compiles and tests green normally.
The per-mutant compile is scoped to the file set from
TestArgs::get_sources_to_compile() (with dynamic_test_linking force-enabled
for mutation). For some selected test contracts this scoped compile returns
Err, and crates/forge/src/mutation/runner.rs maps that Err directly to
MutationResult::Invalid without surfacing the compiler error. The result
is a silent, misleading 0% score with no way to tell a tooling/compile failure
apart from genuinely non-compiling mutants.
To Reproduce
git clone --recurse-submodules https://github.com/morpho-org/morpho-blue
cd morpho-blue
# BROKEN: includes MorphoStorageLibTest -> 92/92 INVALID, score 0.0%
FOUNDRY_PROFILE=no_via_ir FOUNDRY_OPTIMIZER_RUNS=200 \
forge test --mutate src/libraries/SharesMathLib.sol \
--mc "SupplyIntegrationTest|WithdrawIntegrationTest|MorphoStorageLibTest" \
--mutation-timeout 50
# WORKS: drop MorphoStorageLibTest -> mutants killed/survived correctly (~95%)
FOUNDRY_PROFILE=no_via_ir FOUNDRY_OPTIMIZER_RUNS=200 \
forge test --mutate src/libraries/SharesMathLib.sol \
--mc "SupplyIntegrationTest|WithdrawIntegrationTest" \
--mutation-timeout 50
The selected contracts run fine as a normal test (no mutation):
FOUNDRY_PROFILE=no_via_ir forge test \
--mc "SupplyIntegrationTest|WithdrawIntegrationTest|MorphoStorageLibTest"
# -> all green
Other selections also trigger it (e.g. any allowlist including
MorphoStorageLibTest, and at least one of the
Authorization/Callbacks/OnlyOwner integration tests); --no-match-contract
denylists that leave those contracts in the set hit it too. MathLibTest,
MarketParamsLibTest, ExtSloadIntegrationTest, etc. do not trigger it.
Broken output
╭──────────┬───────────┬────────────╮
│ Status ┆ # Mutants ┆ % of Total │
╞══════════╪═══════════╪════════════╡
│ Survived ┆ 0 ┆ 0.0% │
│ Killed ┆ 0 ┆ 0.0% │
│ Invalid ┆ 92 ┆ 100.0% │
│ Skipped ┆ 0 ┆ 0.0% │
╰──────────┴───────────┴────────────╯
Mutation Score: 0.0% (0/0 mutants killed)
Expected behavior
- Mutants should be classified identically regardless of which (compiling,
passing) test contracts are selected; valid mutations must not be reported as
INVALID.
- If a per-mutant compile genuinely fails, the compiler error should be surfaced
(at least under -v/-vvv), so an infra/tooling compile failure is
distinguishable from a non-typechecking mutation.
- A run that produces
0/0 real results should not be presented as a 0.0%
score; it should warn that no mutant was actually evaluated.
Environment
forge 1.7.2-dev (commit eba698dcf, master @ the native-symbolic merge)
- macOS (arm64), solc 0.8.19
Additional context
Relevant code: TestArgs::get_sources_to_compile() in
crates/forge/src/cmd/test/mod.rs and the Err(_) => MutationResult::Invalid
mapping in crates/forge/src/mutation/runner.rs (compile_and_test).
config.dynamic_test_linking is force-enabled for mutation runs, which may be
implicated in the scoped-compile failure.
Component
Forge (
forge test --mutate)Describe the bug
When the test selection passed to mutation testing includes certain test
contracts, every mutant is classified as
INVALID("mutation produced acompilation error") and the run reports a meaningless Mutation Score: 0.0%
(0/0) — even though the exact same mutations are valid and killable with a
slightly different selection, and the project compiles and tests green normally.
The per-mutant compile is scoped to the file set from
TestArgs::get_sources_to_compile()(withdynamic_test_linkingforce-enabledfor mutation). For some selected test contracts this scoped compile returns
Err, andcrates/forge/src/mutation/runner.rsmaps thatErrdirectly toMutationResult::Invalidwithout surfacing the compiler error. The resultis a silent, misleading 0% score with no way to tell a tooling/compile failure
apart from genuinely non-compiling mutants.
To Reproduce
The selected contracts run fine as a normal test (no mutation):
Other selections also trigger it (e.g. any allowlist including
MorphoStorageLibTest, and at least one of theAuthorization/Callbacks/OnlyOwnerintegration tests);--no-match-contractdenylists that leave those contracts in the set hit it too.
MathLibTest,MarketParamsLibTest,ExtSloadIntegrationTest, etc. do not trigger it.Broken output
Expected behavior
passing) test contracts are selected; valid mutations must not be reported as
INVALID.(at least under
-v/-vvv), so an infra/tooling compile failure isdistinguishable from a non-typechecking mutation.
0/0real results should not be presented as a0.0%score; it should warn that no mutant was actually evaluated.
Environment
forge 1.7.2-dev(commiteba698dcf, master @ the native-symbolic merge)Additional context
Relevant code:
TestArgs::get_sources_to_compile()incrates/forge/src/cmd/test/mod.rsand theErr(_) => MutationResult::Invalidmapping in
crates/forge/src/mutation/runner.rs(compile_and_test).config.dynamic_test_linkingis force-enabled for mutation runs, which may beimplicated in the scoped-compile failure.