You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
testfiles: consider reevaluating all files to ensure runtime requirement is met #3141
mandiant/capa-testfiles#322 modified multiple files, some of which were committed a long time ago, which triggered the runtime check to execute. This identified that at least three of the files do not meet our current runtime requirements:
ERROR capa.tests.data: capa ran for 308.1699028015137 check_runtimes.py:61
seconds on
b5f0524e69b3a3cf636c7ac366ca57bf5e3a8fdc8a9f01caf1
96c611a7918a87.elf_, please provide a different
sample so we can test more quickly
ERROR capa.tests.data: capa ran for 374.43834686279297 check_runtimes.py:61
seconds on 92d8ea10ea30e8b534334a1c9857a455.exe_,
please provide a different sample so we can test
more quickly
ERROR capa.tests.data: capa ran for 275.35956621170044 check_runtimes.py:61
seconds on
559efe9f3d4864910d5c5edbaadd3972c6dfc3ba887c48aa67
28b0093898563f.dll_, please provide a different
sample so we can test more quickly
These three files alone account for 15 minutes of runtime. The runtime check only executes for the files modified/added in a given pull request, so there are likely other files that have flown under the radar. IIRC this is only an issue when a thorough test is manually triggered one or more times before a new release. We should reevaluate all test files and replace those that no longer meet our runtime requirements if thorough test runtime becomes a concern.
IIRC this is only an issue when a thorough test is manually triggered one or more times before a new release
and that the file is referenced as an example of a rule.
i think there might be some testfiles that aren't used for rule matching, like possibly for file format tests. i seem to recall we might have added a couple large Go binaries for unusual arch or something to test ELF parsing, realizing that they'd never trigger long CI runs.
To start, I can script an audit running check_runtimes.py across the testfiles to identify all files exceeding the 4-minute threshold, cross-reference which ones are actively tied to rule examples versus test fixtures, and report the list here. From there, we can look into finding or compiling smaller replacement samples for the worst offenders.
Activity
williballenthin commentedon Aug 11, 2026
and that the file is referenced as an example of a rule.
i think there might be some testfiles that aren't used for rule matching, like possibly for file format tests. i seem to recall we might have added a couple large Go binaries for unusual arch or something to test ELF parsing, realizing that they'd never trigger long CI runs.
williballenthin commentedon Aug 11, 2026
anyways, this remains a good idea
ninadgowdru commentedon Oct 25, 2026
Hi @mike-hunhoff @williballenthin,
I'd be interested in tackling this!
To start, I can script an audit running check_runtimes.py across the testfiles to identify all files exceeding the 4-minute threshold, cross-reference which ones are actively tied to rule examples versus test fixtures, and report the list here. From there, we can look into finding or compiling smaller replacement samples for the worst offenders.
Does that sound like a good starting point?