You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
consume direct: block-test failures against Nethermind nethtest are never detected #3609
consume direct --bin=nethtest cannot report a failing blockchain_test against Nethermind.
Two facts combine:
nethtest --blockTest always exits 0 and reports results only as human-readable <name> PASS/FAIL lines on stdout (src/Nethermind/Nethermind.Test.Runner/Program.cs, Run returns 0 unconditionally; BlockchainTestsRunner.cs prints per-test lines, no JSON summary). State tests do emit a JSON array, block tests do not.
packages/testing/src/execution_testing/client_clis/clis/nethermind.py decides the block-test outcome from result.returncode alone and never inspects stdout.
Reproduce: take any blockchain_test fixture, change a surviving storage value in its pre so the executed root no longer matches the RLP-committed root, run uv run consume direct --input <dir> --bin=nethtest -m blockchain_test. Nethermind itself prints FAIL (or throws InvalidBlockException), the harness reports the test as passed.
Verified with Nethermind built from current master (dotnet build of Nethermind.Test.Runner).
Activity
SIDDHANTCOOKIE commentedon Sep 22, 2026
Opened #3634 fixing this — would appreciate a review.
fix(test-consume): run and check nethtest block tests in consume dire…
fix(test-consume): run and check nethtest block tests in consume dire…