Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions config/netbox-common.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ NETBOX_ENRICHMENT=false
NETBOX_DEFAULT_SITE=Malcolm
# Whether or not unobserved network entities in Logstash data will be used to populate NetBox
NETBOX_AUTO_POPULATE=false
# Inherit a new host IP's mask from a unique, site-consistent NetBox prefix, including routed/untagged observations.
# False preserves /32 for bare IPv4 hosts and /128 for bare IPv6 hosts; existing IPs are never rewritten.
NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH=false
# Comma-separated list of CIDR subnets that control which IP addresses are eligible for NetBox autopopulation.
# See "Subnets considered for autopopulation" in the documentation for examples.
# Behavior:
Expand Down
12 changes: 12 additions & 0 deletions docs/asset-interaction-analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,18 @@ Users may select between two behaviors for when the match threshold is not met (
* `NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=false` - the autopopulated device will be created with the manufacturer set to `Unspecified`
* `NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true` - the autopopulated device will be created along with a new manufacturer entry in the NetBox database set to the OUI organization name

### <a name="NetBoxIdentityReconciliation"></a>Existing identities and MAC addresses

`NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH=true` enables prefix-length inheritance for newly created host addresses. For example, a bare observed `10.10.0.7` within a unique, site-consistent `10.10.0.0/24` prefix is created as `10.10.0.7/24`. The most specific containing prefix must be unambiguous, belong to the selected site (directly, through its location, or through its VLAN/site-scoped VLAN group), have no unresolved VRF context, and not be a container prefix. IPv6 is supported. Otherwise, bare addresses retain the `/32` IPv4 or `/128` IPv6 fallback. Routed and untagged observations can inherit the mask: their capture VLAN describes the observation link and need not be the endpoint's native VLAN. This does not relax the separate VLAN requirement for MAC identification or assignment. The option defaults to `false`, preserves explicit input masks, and does not rewrite existing IP records. The inherited mask represents an opt-in inventory policy, not a mask discovered in the packet header.

Passive discovery checks existing host IP records before creating a device. NetBox host addresses are matched independently of their stored masks: an existing `10.10.0.7/24` is not replaced by another `/32`. An unassigned or out-of-scope host address produces `identity_status: unassigned_or_out_of_scope_ip_review` instead of a new device. Existing duplicate host records are marked with `ip_record_conflict`; this does not merge or reassign them. Addresses found only in interface descriptions are not used for automatic matching.

A unique existing MAC object can identify a native device interface when the observed VLAN matches the most specific containing prefix and that prefix/VLAN belongs to the selected site. Site and location scopes and site-scoped VLAN groups are supported. MAC matching is skipped for missing/ambiguous VLANs, ambiguous prefixes or VRFs requiring additional context. This avoids treating a routed next-hop MAC as the IP endpoint's MAC. A local MAC match is labelled `identity_source: local_mac`; `ip_assignment_conflict` indicates that NetBox's IP assignment still needs reconciliation. It does not assign an IP to the imported interface. Conflicting native IP/MAC identities require review. Hostname matches can prevent another candidate from being created but do not establish identity.

For NetBox 4.2 and later, MAC addresses are created through the separate MAC-address API and assigned to auto-created device/VM interfaces. An unambiguous auto-created interface can also receive a missing MAC on a subsequent local observation. Existing MAC objects are not moved between interfaces or claimed from the unassigned pool, and existing primary MACs are preserved. Imported interfaces are not modified. A blank primary MAC field does not establish that an interface's `mac_addresses` list is empty.

Device-lookup caches include MAC, VLAN and hostname context so a local match is not reused for a different observation of the same IP. Creation is serialized per site/host within one Logstash process; this is not a distributed lock across multiple Malcolm deployments. Existing duplicates require a separate review. Enrichment applies to newly processed records and uses current inventory, so historical IP validity and physical/equipment ownership still require independent evidence.

## <a name="NetBoxPopActive"></a>Populate NetBox inventory via active discovery

See [idaholab/Malcolm#136](https://github.com/idaholab/Malcolm/issues/136).
Expand Down
1 change: 1 addition & 0 deletions docs/malcolm-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ Although the configuration script automates many of the following configuration
- `NETBOX_DEFAULT_SITE` - specifies the default NetBox [site name](https://netboxlabs.com/docs/netbox/features/facilities/#sites) for use when [enriching network traffic metadata via NetBox lookups](asset-interaction-analysis.md#NetBoxEnrichment) if a specific site is not otherwise specified for the source of the data (default `Malcolm`)
- `NETBOX_AUTO_POPULATE` - if set to `true`, Logstash will [populate the NetBox inventory](asset-interaction-analysis.md#NetBoxPopPassive) based on observed network traffic
- `NETBOX_AUTO_POPULATE_SUBNETS` - a comma-separated list of private CIDR subnets to control NetBox IP autopopulation (see [**Subnets considered for autopopulation**](asset-interaction-analysis.md#NetBoxAutoPopSubnets); default is an empty string, meaning all private IPv4 and IPv6 ranges are autopopulated)
- `NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH` - if set to `true`, newly created host IPs inherit the length of a unique containing NetBox prefix consistent with the selected site, including routed or untagged observations. Missing/ambiguous site or VRF context and container prefixes retain `/32` for IPv4 or `/128` for IPv6; explicit input masks and existing IP records are preserved (default `false`; see [existing identities and MAC addresses](asset-interaction-analysis.md#NetBoxIdentityReconciliation))
- `NETBOX_AUTO_CREATE_PREFIX` - if set to `true`, Logstash will automatically create private subnet prefixes in the [NetBox inventory](asset-interaction-analysis.md#NetBoxPopPassive) based on observed network traffic
- `NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER` - if set to `true`, new manufacturer entries will be created in the NetBox database when [matching device manufacturers to OUIs](asset-interaction-analysis.md#NetBoxPopPassiveOUIMatch) (default `true`)
- `NETBOX_DEFAULT_FUZZY_THRESHOLD` - fuzzy-matching threshold for [matching device manufacturers to OUIs](asset-interaction-analysis.md#NetBoxPopPassiveOUIMatch) (default `0.95`)
Expand Down
10 changes: 10 additions & 0 deletions logstash/pipelines/enrichment/21_netbox.conf
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ filter {
"cache_size_env" => "NETBOX_CACHE_SIZE"
"cache_ttl_env" => "NETBOX_CACHE_TTL"
"autopopulate_env" => "NETBOX_AUTO_POPULATE"
"autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH"
"autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS"
"default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER"
"default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE"
Expand Down Expand Up @@ -192,6 +193,7 @@ filter {
"cache_size_env" => "NETBOX_CACHE_SIZE"
"cache_ttl_env" => "NETBOX_CACHE_TTL"
"autopopulate_env" => "NETBOX_AUTO_POPULATE"
"autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH"
"autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS"
"default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER"
"default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE"
Expand Down Expand Up @@ -223,6 +225,7 @@ filter {
"cache_size_env" => "NETBOX_CACHE_SIZE"
"cache_ttl_env" => "NETBOX_CACHE_TTL"
"autopopulate_env" => "NETBOX_AUTO_POPULATE"
"autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH"
"autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS"
"default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER"
"default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE"
Expand Down Expand Up @@ -251,6 +254,7 @@ filter {
"cache_size_env" => "NETBOX_CACHE_SIZE"
"cache_ttl_env" => "NETBOX_CACHE_TTL"
"autopopulate_env" => "NETBOX_AUTO_POPULATE"
"autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH"
"autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS"
"default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER"
"default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE"
Expand Down Expand Up @@ -285,6 +289,7 @@ filter {
"cache_size_env" => "NETBOX_CACHE_SIZE"
"cache_ttl_env" => "NETBOX_CACHE_TTL"
"autopopulate_env" => "NETBOX_AUTO_POPULATE"
"autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH"
"autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS"
"default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER"
"default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE"
Expand Down Expand Up @@ -313,6 +318,7 @@ filter {
"cache_size_env" => "NETBOX_CACHE_SIZE"
"cache_ttl_env" => "NETBOX_CACHE_TTL"
"autopopulate_env" => "NETBOX_AUTO_POPULATE"
"autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH"
"autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS"
"default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER"
"default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE"
Expand Down Expand Up @@ -349,6 +355,7 @@ filter {
"cache_size_env" => "NETBOX_CACHE_SIZE"
"cache_ttl_env" => "NETBOX_CACHE_TTL"
"autopopulate_env" => "NETBOX_AUTO_POPULATE"
"autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH"
"autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS"
}
}
Expand Down Expand Up @@ -376,6 +383,7 @@ filter {
"cache_size_env" => "NETBOX_CACHE_SIZE"
"cache_ttl_env" => "NETBOX_CACHE_TTL"
"autopopulate_env" => "NETBOX_AUTO_POPULATE"
"autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH"
"autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS"
"default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER"
"default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE"
Expand Down Expand Up @@ -409,6 +417,7 @@ filter {
"cache_size_env" => "NETBOX_CACHE_SIZE"
"cache_ttl_env" => "NETBOX_CACHE_TTL"
"autopopulate_env" => "NETBOX_AUTO_POPULATE"
"autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH"
"autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS"
}
}
Expand Down Expand Up @@ -438,6 +447,7 @@ filter {
"cache_size_env" => "NETBOX_CACHE_SIZE"
"cache_ttl_env" => "NETBOX_CACHE_TTL"
"autopopulate_env" => "NETBOX_AUTO_POPULATE"
"autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH"
"autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS"
"default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER"
"default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE"
Expand Down
Loading