Skip to content

Fix NetBox discovery matching, MAC creation, and optional host masks - #1296

Open
berlikm wants to merge 5 commits into
cisagov:contributor-prsfrom
berlikm:codex/netbox-existing-hosts-and-macs
Open

berlikm wants to merge 5 commits into
cisagov:contributor-prsfrom
berlikm:codex/netbox-existing-hosts-and-macs

Conversation

@berlikm

@berlikm berlikm commented Nov 6, 2026 •

Copy link
Copy Markdown

Passive NetBox discovery can create a duplicate device and /32 when a host already exists as an unassigned IP record, ignore existing native MAC objects, and post the read-only interface mac_address field used before NetBox 4.2. An opt-in policy is also needed to use known network-prefix lengths for newly created host addresses.

The change guards existing host IPs before creating candidates, resolves unique native MAC identities in a matching prefix/VLAN/site context, and uses the current MAC-address API for eligible auto-created device/VM interfaces. Imported assignments and existing primary MACs are preserved. Contextual cache keys and process-local site/host locks prevent lookup leakage and concurrent duplicate creation.

NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH defaults to false. When enabled, new host IPs inherit the most specific unique containing IPAM prefix's length only where site and VRF context are consistent. Site resolution supports direct site/location scope or a prefix's VLAN/site-scoped VLAN group. Container prefixes, missing/ambiguous scope and unresolved VRFs retain /32 or /128. Explicit masks and existing IP records are preserved. This is inventory policy, not a mask measured from packet headers.

Mask policy and MAC ownership use separate rules. A routed endpoint may be observed on another VLAN, and a first-observation log such as Zeek known_hosts may omit VLAN metadata entirely. Both can still use an authoritative scoped IPAM mask. For example, an untagged first event for 10.10.0.7, followed by traffic on a different capture VLAN, can create one 10.10.0.7/24 from the known prefix. Matching capture/native VLAN context remains mandatory for MAC lookup/assignment, so this does not assign gateway MACs to endpoints. The option is wired into the enrichment pipeline and documented in the configuration reference and example environment file.

Validation:

  • 49 regression checks executed against the production Ruby source under JRuby, including unassigned-IP guards, routing/MAC conflicts, pagination, idempotency, concurrent creation, cache updates, IPv6, longest-prefix selection, first-untagged-then-routed observations, site-only prefixes, container rejection and API failure before device creation.
  • Twelve read-only checks against NetBox 4.5.10 verified real prefix selection for local/routed/untagged observations, explicit-mask preservation, configuration parsing/defaults, and continued rejection of routed or untagged MAC ownership.
  • Real NetBox 4.5.10 API tests for the identity/MAC fix verified a temporary IPv4 device and IPv6 VM, MAC objects, primary MAC/IP assignments, repeat MAC updates and cleanup.
  • Pilot deployment verified the enabled flag, all ten active filter parameters, source fingerprints and healthy ingestion. A temporary API-only fixture using the deployed source and actual environment created /24 without an observed VLAN, accepted the same scoped mask on a routed observation, rejected the gateway MAC, verified primary IP and removed all temporary fixtures.

Existing duplicate records and historical assignments require separate reconciliation. MAC attribution remains limited by local observation coverage. Synchronization is local to one Logstash process rather than a distributed lock across multiple Malcolm deployments.

mmguero and others added 3 commits November 3, 2026 17:55
Adds docs/contributing-security-model.md, describing how Malcolm's
components trust one another, where its security boundaries are, and
which configuration choices move them.

Covers:

- the internal container network as a single trust domain, and why
  internal services trust their reverse proxy
- what the installer publishes to the outside, under which "Expose
  Malcolm Service Ports" selections, and that exposed services bind
  0.0.0.0 rather than loopback
- the aggregator/sensor reachback enclave
- the host boundary: the PUSER_PRIV_DROP privilege-drop model, the
  difference between dropping at the entrypoint and dropping per
  supervisord program, and which processes legitimately remain root
- nginx as the authentication layer, the identity headers backends
  consume, what RBAC does and does not separate, and the RBAC URI
  normalization problem
- data ingestion and expensive queries as trusted user capabilities
- configuration placeholders substituted at container startup, and
  build-time credential handling
- deployment choices that move the boundaries, and what is out of scope,
  including the pinning and checksum verification the build already does

Also adds SECURITY.md covering vulnerability reporting, what to expect
after a report, and CVE assignment through CISA's VINCE system, and
links the new document from the contributor guide and documentation
index.

Fixes cisagov#1279

Signed-off-by: Seth Grover <seth.d.grover@gmail.com>
@berlikm berlikm changed the title Fix NetBox identity reconciliation and MAC address creation Fix NetBox discovery matching, MAC creation, and optional host masks Nov 6, 2026
@mmguero
mmguero changed the base branch from main to contributor-prs November 6, 2026 15:02
@mmguero

mmguero commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

@berlikm thanks for the contribution! I've just glanced at it and will get to a more in-depth triage/review hopefully in the not-to-distance future. At the moment we're a little bit resource-constrained and swimming in PRs, so please have patience with us and we'll try to get this slated for either this month's or next month's release. Again, we appreciate you pitching in!

Preserve upstream TLS verification and same-origin API URL handling alongside endpoint reconciliation and host-mask policy.
@mmguero mmguero added this to Malcolm Nov 9, 2026
@mmguero mmguero added this to the z.staging milestone Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants