Skip to content

testfiles: consider reevaluating all files to ensure runtime requirement is met #3141

Open

Description

@mike-hunhoff

mandiant/capa-testfiles#322 modified multiple files, some of which were committed a long time ago, which triggered the runtime check to execute. This identified that at least three of the files do not meet our current runtime requirements:

ERROR    capa.tests.data: capa ran for 308.1699028015137    check_runtimes.py:61
         seconds on                                                             
         b5f0524e69b3a3cf636c7ac366ca57bf5e3a8fdc8a9f01caf1                     
         96c611a7918a87.elf_, please provide a different                        
         sample so we can test more quickly  

ERROR    capa.tests.data: capa ran for 374.43834686279297   check_runtimes.py:61
         seconds on 92d8ea10ea30e8b534334a1c9857a455.exe_,                      
         please provide a different sample so we can test                       
         more quickly     

ERROR    capa.tests.data: capa ran for 275.35956621170044   check_runtimes.py:61
         seconds on                                                             
         559efe9f3d4864910d5c5edbaadd3972c6dfc3ba887c48aa67                     
         28b0093898563f.dll_, please provide a different                        
         sample so we can test more quickly   

These three files alone account for 15 minutes of runtime. The runtime check only executes for the files modified/added in a given pull request, so there are likely other files that have flown under the radar. IIRC this is only an issue when a thorough test is manually triggered one or more times before a new release. We should reevaluate all test files and replace those that no longer meet our runtime requirements if thorough test runtime becomes a concern.

Activity

  1. added
    enhancementNew feature or request
    CIContinuous Integration configuration
    on Aug 11, 2026
  2. williballenthin commented on Aug 11, 2026

    @williballenthin
    Collaborator

    IIRC this is only an issue when a thorough test is manually triggered one or more times before a new release

    and that the file is referenced as an example of a rule.

    i think there might be some testfiles that aren't used for rule matching, like possibly for file format tests. i seem to recall we might have added a couple large Go binaries for unusual arch or something to test ELF parsing, realizing that they'd never trigger long CI runs.

  3. williballenthin commented on Aug 11, 2026

    @williballenthin
    Collaborator

    anyways, this remains a good idea

  4. ninadgowdru commented on Oct 25, 2026

    @ninadgowdru

    Hi @mike-hunhoff @williballenthin,

    I'd be interested in tackling this!

    To start, I can script an audit running check_runtimes.py across the testfiles to identify all files exceeding the 4-minute threshold, cross-reference which ones are actively tied to rule examples versus test fixtures, and report the list here. From there, we can look into finding or compiling smaller replacement samples for the worst offenders.

    Does that sound like a good starting point?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CIContinuous Integration configurationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions