Skip to content

freeze format may emit import features with key "import_" rather than "import" #3085

Open

Description

@williballenthin

need to verify, but this would be unintended. we want they key to be import and then map back to import_ when in python-land.

Activity

  1. williballenthin commented on May 20, 2026

    @williballenthin
    Author
  2. williballenthin commented on May 20, 2026

    @williballenthin
    CollaboratorAuthor

    What happens: model_dump_json() is called without by_alias=True on lines 457 and 576 of capa/features/freeze/init.py. Pydantic's default is to serialize using the Python field name, so the JSON key is "import_" (with trailing underscore) instead of "import".

    Why it exists: The Python field is named import_ because import is a reserved keyword. The Field(alias="import") is configured correctly, and populate_by_name=True means deserialization accepts both "import" and "import_" — so roundtrip tests pass even though the output key is wrong. This masks the bug.

    The same issue affects other aliased fields: FunctionNameFeature.function_name (alias "function name"), ClassFeature.class_ (alias "class"), OperandNumberFeature.operand_number (alias "operand number"), OperandOffsetFeature.operand_offset (alias "operand offset"), and several fields in init.py like base_address (alias "base address"), basic_block/basic_blocks, global_.

    Recommendation: The cleanest fix is to add by_alias=True to both model_dump_json() calls (lines 457 and 576), plus all the sorting lambdas that also call model_dump_json(). This ensures the freeze file uses the intended human-readable keys ("import", "function name", "class", etc.) rather than Python-internal names.

    Alternatively, you could set model_config = ConfigDict(frozen=True, populate_by_name=True, serialize_by_alias=True) on FeatureModel (Pydantic v2 feature) to make alias serialization the default for all models inheriting from it — but that's a broader change that could affect result documents and other serialization paths. The targeted by_alias=True at the two freeze dump sites is safer and more explicit.

  3. williballenthin commented on May 20, 2026

    @williballenthin
    CollaboratorAuthor

    Impact of serialize_by_alias=True on FeatureModel

    The change would be on FeatureModel only (line 25 of features.py). The container models in init.py and result_document.py inherit from BaseModel/HashableModel/FrozenModel — not FeatureModel — so they are not directly affected. However, when a parent model serializes a nested FeatureModel field, Pydantic respects the child's serialize_by_alias config.

    Affected fields (5 total, all in features.py)

    ┌──────────────────────┬────────────────┬──────────────────┬──────────────────┐
    │        Model         │     Field      │ Current JSON key │   Would become   │
    ├──────────────────────┼────────────────┼──────────────────┼──────────────────┤
    │ ImportFeature        │ import_        │ "import_"        │ "import"         │
    ├──────────────────────┼────────────────┼──────────────────┼──────────────────┤
    │ FunctionNameFeature  │ function_name  │ "function_name"  │ "function name"  │
    ├──────────────────────┼────────────────┼──────────────────┼──────────────────┤
    | ClassFeature         │ class_         │ "class_"         │ "class"          │
    ├──────────────────────┼────────────────┼──────────────────┼──────────────────┤
    │ OperandNumberFeature │ operand_number │ "operand_number" │ "operand number" │
    ├──────────────────────┼────────────────┼──────────────────┼──────────────────┤
    │ OperandOffsetFeature │ operand_offset │ "operand_offset" │ "operand offset" │
    └──────────────────────┴────────────────┴──────────────────┴──────────────────┘
    

    Serialization sites impacted

    1. Freeze format output (capa/features/freeze/init.py)
    • Lines 457, 576: freeze.model_dump_json() — the two main dump points. Nested Feature fields would serialize using aliases. This is the desired fix.
    • 10 sorting lambdas (lines 362, 372, 385, 398, 411, 473, 483, 497, 510, 524): feature.model_dump_json() — sort keys would change, but that's cosmetic (affects reproducible output ordering, not correctness).
    1. Feature comparison (capa/features/common.py:161-162)
    • Feature.lt uses model_dump_json() for ordering. Key names would change, altering sort order of features. Functionally fine — only needs to be consistent, not canonical.
    1. Result document JSON output (nested Feature fields)
    • capa/render/json.py:21 — ResultDocument.model_dump_json()
    • capa/main.py:1013 — result_doc.model_dump_json()
    • capa/ida/plugin/form.py:1309 — resdoc_cache.model_dump_json()
    • scripts/bulk-process.py:148 — doc.model_dump()
    • scripts/bulk-process.py:215, scripts/proto-to-results.py:75 — more ResultDocument dumps
    • tests/test_result_document.py:252,260,267 — test assertions on JSON output

    In all these, Feature models are nested inside StatementNode.child and FeatureNode.feature. The Feature portion of the JSON would change from "import_" to "import", etc. This is also the correct/desired behavior — but it means the result document JSON schema changes too, not just freeze files.

    1. Already using by_alias=True (would become redundant but harmless)
    • capa/render/vverbose.py:167 — child.model_dump(by_alias=True)
    • capa/render/vverbose.py:227 — feature.model_dump(by_alias=True)
    • capa/ida/plugin/model.py:560 — feature.dict(by_alias=True)
    • tests/test_result_document.py:319 — meta.model_dump(by_alias=True) (this one is on a Metadata model, not FeatureModel, so unrelated)
    1. Manual workarounds that could be removed
    • capa/render/vverbose.py:220-224 — special-cases ImportFeature.import_ and ClassFeature.class_ because the alias-based lookup wouldn't work without by_alias. With serialize_by_alias=True, the generic path on line 227 would handle these correctly.

    Bottom line

    The FeatureModel ConfigDict approach is clean and correct — it fixes both freeze files and result document JSON in one shot. The blast radius is well-contained: only the 5 aliased fields change their serialization keys, and everywhere that consumes these already expects the alias form (that's why the aliases exist). The by_alias=True calls become redundant but harmless, and the manual workarounds in vverbose.py become removable dead code.

  4. added
    bugSomething isn't working
    breaking-changeintroduces a breaking change that should be released in a major version
    on May 20, 2026
  5. williballenthin commented on May 20, 2026

    @williballenthin
    CollaboratorAuthor
    1. HashableModel/BaseModel subclasses in capa/features/freeze/init.py

    5 aliased fields on container models:

      ┌───────────────────────────────┬──────────────────┬────────────────┐
      │             Field             │ Current JSON key │  Intended key  │
      ├───────────────────────────────┼──────────────────┼────────────────┤
      │ BasicBlockFeature.basic_block │ "basic_block"    │ "basic block"  │
      ├───────────────────────────────┼──────────────────┼────────────────┤
      │ FunctionFeatures.basic_blocks │ "basic_blocks"   │ "basic blocks" │
      ├───────────────────────────────┼──────────────────┼────────────────┤
      │ StaticFeatures.global_        │ "global_"        │ "global"       │
      ├───────────────────────────────┼──────────────────┼────────────────┤
      │ DynamicFeatures.global_       │ "global_"        │ "global"       │
      ├───────────────────────────────┼──────────────────┼────────────────┤
      │ Freeze.base_address           │ "base_address"   │ "base address" │
      └───────────────────────────────┴──────────────────┴────────────────┘
    

    Serialization paths: same freeze model_dump_json() calls. These inherit from HashableModel/BaseModel, not FeatureModel, so the FeatureModel ConfigDict fix alone wouldn't help — these need their own fix.

    1. FrozenModel subclasses in capa/render/result_document.py

    7 aliased fields across two models:

      ┌─────────────────────────────────────┬──────────────────────────┬──────────────────────────┐
      │                Field                │     Current JSON key     │       Intended key       │
      ├─────────────────────────────────────┼──────────────────────────┼──────────────────────────┤
      │ MaecMetadata.analysis_conclusion    │ "analysis_conclusion"    │ "analysis-conclusion"    │
      ├─────────────────────────────────────┼──────────────────────────┼──────────────────────────┤
      │ MaecMetadata.analysis_conclusion_ov │ "analysis_conclusion_ov" │ "analysis-conclusion-ov" │
      ├─────────────────────────────────────┼──────────────────────────┼──────────────────────────┤
      │ MaecMetadata.malware_family         │ "malware_family"         │ "malware-family"         │
      ├─────────────────────────────────────┼──────────────────────────┼──────────────────────────┤
      │ MaecMetadata.malware_category       │ "malware_category"       │ "malware-category"       │
      ├─────────────────────────────────────┼──────────────────────────┼──────────────────────────┤
      │ MaecMetadata.malware_category_ov    │ "malware_category_ov"    │ "malware-category-ov"    │
      ├─────────────────────────────────────┼──────────────────────────┼──────────────────────────┤
      │ RuleMetadata.attack                 │ "attack"                 │ "att&ck"                 │
      ├─────────────────────────────────────┼──────────────────────────┼──────────────────────────┤
      │ RuleMetadata.is_subscope_rule       │ "is_subscope_rule"       │ "capa/subscope-rule"     │
      └─────────────────────────────────────┴──────────────────────────┴──────────────────────────┘
    

    Serialization paths: capa/render/json.py:21, capa/main.py:1013, capa/ida/plugin/form.py:1309, scripts/bulk-process.py:148,215, scripts/proto-to-results.py:75.

  6. williballenthin commented on May 20, 2026

    @williballenthin
    CollaboratorAuthor

    The web UI is hardcoded to the current (buggy) Python field names:

    • rdocParser.js:508 — feature.operand_offset (not feature["operand offset"])
    • rdocParser.js:510 — feature.class_ (not feature["class"])
    • rdocParser.js:512 — feature.import_ (not feature["import"])
    • rdocParser.js:115 — rule.meta.attack (not rule.meta["att&ck"])
    • rdocParser.js:268 — finfo.matched_basic_blocks (not finfo["matched basic blocks"])
    • Test fixtures in rdocParser.test.js also use matched_basic_blocks
  7. williballenthin commented on Nov 4, 2026

    @williballenthin
    CollaboratorAuthor

    please do not volunteer PRs without thoroughly discussing the issue with maintainers. writing the code is not hard, especially for an LLM. the difficult part is coordinating the breaking changes with the ecosystem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    breaking-changeintroduces a breaking change that should be released in a major versionbugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions