Skip to content

Skip completed NetBox device library imports and support vendor filtering - #1196

Merged
mmguero merged 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/406-skip-redundant-netbox-library-import
Oct 13, 2026
Merged

mmguero merged 1 commit into
cisagov:contributor-prsfrom
sylvesterkaczmarek:fix/406-skip-redundant-netbox-library-import

Conversation

@sylvesterkaczmarek

@sylvesterkaczmarek sylvesterkaczmarek commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Refs #406 (NetBox device-type library startup performance). The full first-time importer performance issue remains open.

Changes

  • Skip repeated full device-type library imports after a successful import, using a source-and-vendor fingerprint stored as a NetBox database tag rather than in transient container storage.
  • Clean up superseded malcolm-device-library-* tags after writing the current completion marker; log cleanup failures without invalidating a successful import.
  • Normalize vendor names with the upstream importer's slug rules, so equivalent spellings use one fingerprint. Support an opt-in vendor filter and forced refresh.
  • Reject caching for failed or partial imports. Compared the output checks against the actual mmguero-dev/Device-Type-Library-Import importer failure summary messages.
  • Document that vendor filtering also limits manufacturer coverage for MAC OUI matching.

Validation

  • Rebased onto cisagov/Malcolm:contributor-prs at 7aa2dddd7ce8ed00ffcfd7c4db9e14cdfcfefffd.
  • Moved the direct-import unit tests into shared/tests/test_netbox_device_type_import_cache.py and added the NetBox scripts directory to shared test search paths.
  • 14 targeted tests and 7 subtests passed, including stale-marker cleanup, cleanup failures, equivalent vendor spellings, import summaries, and import retry/skip behavior.
  • Ruff lint and formatting, Python syntax, and Git diff checks pass.
  • Live NetBox/PostgreSQL import timings and a full deployment run were not performed. The suggested --only-new benchmark remains optional and unverified.

@mmguero mmguero added netbox Related to Malcolm's use of NetBox performance Related to speed/performance labels Oct 12, 2026
@mmguero mmguero added this to Malcolm Oct 12, 2026
@mmguero

mmguero commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator

Thanks for this. Keeping the marker in NetBox's database is the right call, and the fail-open handling around the marker lookup and write looks good. I checked the pieces it depends on. control-scripts/* is copied wholesale in netbox.Dockerfile, so the new module ships. run_process captures stderr, so the success check sees the importer's error output. And upstream's --vendors takes nargs="+" and also splits on commas, so the command line you build works. A few changes I'm requesting before merging:

Clean up old marker tags. The library and the importer are both pulled from unpinned branches at build time, so the fingerprint changes on every release build. Each upgrade leaves another malcolm-device-library-* tag in NetBox's tag list. When record_completion creates a new marker, please have it delete any other tags with the malcolm-device-library- prefix. If the delete fails, log a warning and keep going, like the other marker operations.

Match upstream's vendor normalization. The importer reduces each vendor to slug form with re.sub(r"\W+", "-", v.strip().casefold()), so Hewlett Packard Enterprise and hewlett-packard-enterprise select the same vendor. selected_vendors only strips and casefolds, so those two spellings give different fingerprints and cause an unneeded full import. Please apply the same re.sub in selected_vendors so the marker reflects what the importer actually selects.

Check the success regex against the importer's real output. import_successful looks for failure lines in the importer's output. Please confirm the patterns match the messages core/import_run.py actually prints for failed and partially updated types, and include an example line from it in the test cases. If the wording doesn't match, a partial import gets cached and won't be retried until the library changes or someone sets NETBOX_DEVICETYPE_IMPORT_FORCE.

Docs: note the effect on OUI matching. Malcolm's enrichment fuzzy-matches MAC OUI organization names against NetBox's manufacturer list. Limiting the import to a few vendors also limits which manufacturers can match, so devices from other vendors either won't match or will have manufacturers autocreated with no device types. Please add a sentence about this to the paragraph in asset-interaction-analysis.md.

Issue linkage. Please change Addresses #406 to Refs #406. This speeds up restarts and lets you shrink the first import, but the full first-time import is still slow and still needs investigating upstream, so I want to keep #406 open for that.

Optional, worth evaluating: the importer has an --only-new flag that creates missing types and skips existing ones without comparing them. On a restart against an already-populated NetBox, that might be fast enough to compare against the marker approach, or to use alongside it for the case where the marker is missing. If you have a NetBox handy, timing a restart in default mode against --only-new would tell us whether it's worth adding. That's not a blocker for this PR.

Signed-off-by: Sylvester Kaczmarek <16242628+sylvesterkaczmarek@users.noreply.github.com>
@sylvesterkaczmarek
sylvesterkaczmarek force-pushed the fix/406-skip-redundant-netbox-library-import branch from 21cc494 to ec663c7 Compare October 13, 2026 10:44
@sylvesterkaczmarek
sylvesterkaczmarek changed the base branch from main to contributor-prs October 13, 2026 10:45
@sylvesterkaczmarek

Copy link
Copy Markdown
Contributor Author

Updated in ec663c7 on contributor-prs. A new marker now removes superseded NetBox library tags, while cleanup failures only produce warnings. Vendor fingerprints follow the upstream slug normalization. I checked the partial and failed import regex against the actual messages in mmguero-dev/Device-Type-Library-Import core/import_run.py and added those lines to the tests. The OUI documentation now explains restricted-vendor effects. Tests are in shared/tests, with 14 targeted tests and 7 subtests passing. I did not run a live NetBox benchmark; the optional --only-new comparison remains for future investigation.

@mmguero mmguero self-assigned this Oct 13, 2026
@mmguero mmguero moved this to Review in Malcolm Oct 13, 2026
@mmguero mmguero modified the milestones: v26.09.0, v26.10.0 Oct 13, 2026
@mmguero
mmguero merged commit 0bf997d into cisagov:contributor-prs Oct 13, 2026
@github-project-automation github-project-automation Bot moved this from Review to Done in Malcolm Oct 13, 2026
@mmguero mmguero mentioned this pull request Oct 29, 2026
@mmguero mmguero moved this from Done to Released in Malcolm Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

netbox Related to Malcolm's use of NetBox performance Related to speed/performance

Projects

Status: Released

Development

Successfully merging this pull request may close these issues.

2 participants