diff --git a/config/netbox-common.env.example b/config/netbox-common.env.example index effe2d20b..035a027f6 100644 --- a/config/netbox-common.env.example +++ b/config/netbox-common.env.example @@ -7,6 +7,9 @@ NETBOX_ENRICHMENT=false NETBOX_DEFAULT_SITE=Malcolm # Whether or not unobserved network entities in Logstash data will be used to populate NetBox NETBOX_AUTO_POPULATE=false +# Inherit a new host IP's mask from a unique, site-consistent NetBox prefix, including routed/untagged observations. +# False preserves /32 for bare IPv4 hosts and /128 for bare IPv6 hosts; existing IPs are never rewritten. +NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH=false # Comma-separated list of CIDR subnets that control which IP addresses are eligible for NetBox autopopulation. # See "Subnets considered for autopopulation" in the documentation for examples. # Behavior: diff --git a/docs/asset-interaction-analysis.md b/docs/asset-interaction-analysis.md index 6a53b72e4..43d2ac8e1 100644 --- a/docs/asset-interaction-analysis.md +++ b/docs/asset-interaction-analysis.md @@ -155,6 +155,18 @@ Users may select between two behaviors for when the match threshold is not met ( * `NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=false` - the autopopulated device will be created with the manufacturer set to `Unspecified` * `NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER=true` - the autopopulated device will be created along with a new manufacturer entry in the NetBox database set to the OUI organization name +### Existing identities and MAC addresses + +`NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH=true` enables prefix-length inheritance for newly created host addresses. For example, a bare observed `10.10.0.7` within a unique, site-consistent `10.10.0.0/24` prefix is created as `10.10.0.7/24`. The most specific containing prefix must be unambiguous, belong to the selected site (directly, through its location, or through its VLAN/site-scoped VLAN group), have no unresolved VRF context, and not be a container prefix. IPv6 is supported. Otherwise, bare addresses retain the `/32` IPv4 or `/128` IPv6 fallback. Routed and untagged observations can inherit the mask: their capture VLAN describes the observation link and need not be the endpoint's native VLAN. This does not relax the separate VLAN requirement for MAC identification or assignment. The option defaults to `false`, preserves explicit input masks, and does not rewrite existing IP records. The inherited mask represents an opt-in inventory policy, not a mask discovered in the packet header. + +Passive discovery checks existing host IP records before creating a device. NetBox host addresses are matched independently of their stored masks: an existing `10.10.0.7/24` is not replaced by another `/32`. An unassigned or out-of-scope host address produces `identity_status: unassigned_or_out_of_scope_ip_review` instead of a new device. Existing duplicate host records are marked with `ip_record_conflict`; this does not merge or reassign them. Addresses found only in interface descriptions are not used for automatic matching. + +A unique existing MAC object can identify a native device interface when the observed VLAN matches the most specific containing prefix and that prefix/VLAN belongs to the selected site. Site and location scopes and site-scoped VLAN groups are supported. MAC matching is skipped for missing/ambiguous VLANs, ambiguous prefixes or VRFs requiring additional context. This avoids treating a routed next-hop MAC as the IP endpoint's MAC. A local MAC match is labelled `identity_source: local_mac`; `ip_assignment_conflict` indicates that NetBox's IP assignment still needs reconciliation. It does not assign an IP to the imported interface. Conflicting native IP/MAC identities require review. Hostname matches can prevent another candidate from being created but do not establish identity. + +For NetBox 4.2 and later, MAC addresses are created through the separate MAC-address API and assigned to auto-created device/VM interfaces. An unambiguous auto-created interface can also receive a missing MAC on a subsequent local observation. Existing MAC objects are not moved between interfaces or claimed from the unassigned pool, and existing primary MACs are preserved. Imported interfaces are not modified. A blank primary MAC field does not establish that an interface's `mac_addresses` list is empty. + +Device-lookup caches include MAC, VLAN and hostname context so a local match is not reused for a different observation of the same IP. Creation is serialized per site/host within one Logstash process; this is not a distributed lock across multiple Malcolm deployments. Existing duplicates require a separate review. Enrichment applies to newly processed records and uses current inventory, so historical IP validity and physical/equipment ownership still require independent evidence. + ## Populate NetBox inventory via active discovery See [idaholab/Malcolm#136](https://github.com/idaholab/Malcolm/issues/136). diff --git a/docs/malcolm-config.md b/docs/malcolm-config.md index 458b40af2..52eebe0ca 100644 --- a/docs/malcolm-config.md +++ b/docs/malcolm-config.md @@ -137,6 +137,7 @@ Although the configuration script automates many of the following configuration - `NETBOX_DEFAULT_SITE` - specifies the default NetBox [site name](https://netboxlabs.com/docs/netbox/features/facilities/#sites) for use when [enriching network traffic metadata via NetBox lookups](asset-interaction-analysis.md#NetBoxEnrichment) if a specific site is not otherwise specified for the source of the data (default `Malcolm`) - `NETBOX_AUTO_POPULATE` - if set to `true`, Logstash will [populate the NetBox inventory](asset-interaction-analysis.md#NetBoxPopPassive) based on observed network traffic - `NETBOX_AUTO_POPULATE_SUBNETS` - a comma-separated list of private CIDR subnets to control NetBox IP autopopulation (see [**Subnets considered for autopopulation**](asset-interaction-analysis.md#NetBoxAutoPopSubnets); default is an empty string, meaning all private IPv4 and IPv6 ranges are autopopulated) + - `NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH` - if set to `true`, newly created host IPs inherit the length of a unique containing NetBox prefix consistent with the selected site, including routed or untagged observations. Missing/ambiguous site or VRF context and container prefixes retain `/32` for IPv4 or `/128` for IPv6; explicit input masks and existing IP records are preserved (default `false`; see [existing identities and MAC addresses](asset-interaction-analysis.md#NetBoxIdentityReconciliation)) - `NETBOX_AUTO_CREATE_PREFIX` - if set to `true`, Logstash will automatically create private subnet prefixes in the [NetBox inventory](asset-interaction-analysis.md#NetBoxPopPassive) based on observed network traffic - `NETBOX_DEFAULT_AUTOCREATE_MANUFACTURER` - if set to `true`, new manufacturer entries will be created in the NetBox database when [matching device manufacturers to OUIs](asset-interaction-analysis.md#NetBoxPopPassiveOUIMatch) (default `true`) - `NETBOX_DEFAULT_FUZZY_THRESHOLD` - fuzzy-matching threshold for [matching device manufacturers to OUIs](asset-interaction-analysis.md#NetBoxPopPassiveOUIMatch) (default `0.95`) diff --git a/logstash/pipelines/enrichment/21_netbox.conf b/logstash/pipelines/enrichment/21_netbox.conf index 74146fc21..c0f000cda 100644 --- a/logstash/pipelines/enrichment/21_netbox.conf +++ b/logstash/pipelines/enrichment/21_netbox.conf @@ -160,6 +160,7 @@ filter { "cache_size_env" => "NETBOX_CACHE_SIZE" "cache_ttl_env" => "NETBOX_CACHE_TTL" "autopopulate_env" => "NETBOX_AUTO_POPULATE" + "autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH" "autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS" "default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER" "default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE" @@ -192,6 +193,7 @@ filter { "cache_size_env" => "NETBOX_CACHE_SIZE" "cache_ttl_env" => "NETBOX_CACHE_TTL" "autopopulate_env" => "NETBOX_AUTO_POPULATE" + "autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH" "autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS" "default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER" "default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE" @@ -223,6 +225,7 @@ filter { "cache_size_env" => "NETBOX_CACHE_SIZE" "cache_ttl_env" => "NETBOX_CACHE_TTL" "autopopulate_env" => "NETBOX_AUTO_POPULATE" + "autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH" "autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS" "default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER" "default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE" @@ -251,6 +254,7 @@ filter { "cache_size_env" => "NETBOX_CACHE_SIZE" "cache_ttl_env" => "NETBOX_CACHE_TTL" "autopopulate_env" => "NETBOX_AUTO_POPULATE" + "autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH" "autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS" "default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER" "default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE" @@ -285,6 +289,7 @@ filter { "cache_size_env" => "NETBOX_CACHE_SIZE" "cache_ttl_env" => "NETBOX_CACHE_TTL" "autopopulate_env" => "NETBOX_AUTO_POPULATE" + "autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH" "autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS" "default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER" "default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE" @@ -313,6 +318,7 @@ filter { "cache_size_env" => "NETBOX_CACHE_SIZE" "cache_ttl_env" => "NETBOX_CACHE_TTL" "autopopulate_env" => "NETBOX_AUTO_POPULATE" + "autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH" "autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS" "default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER" "default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE" @@ -349,6 +355,7 @@ filter { "cache_size_env" => "NETBOX_CACHE_SIZE" "cache_ttl_env" => "NETBOX_CACHE_TTL" "autopopulate_env" => "NETBOX_AUTO_POPULATE" + "autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH" "autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS" } } @@ -376,6 +383,7 @@ filter { "cache_size_env" => "NETBOX_CACHE_SIZE" "cache_ttl_env" => "NETBOX_CACHE_TTL" "autopopulate_env" => "NETBOX_AUTO_POPULATE" + "autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH" "autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS" "default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER" "default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE" @@ -409,6 +417,7 @@ filter { "cache_size_env" => "NETBOX_CACHE_SIZE" "cache_ttl_env" => "NETBOX_CACHE_TTL" "autopopulate_env" => "NETBOX_AUTO_POPULATE" + "autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH" "autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS" } } @@ -438,6 +447,7 @@ filter { "cache_size_env" => "NETBOX_CACHE_SIZE" "cache_ttl_env" => "NETBOX_CACHE_TTL" "autopopulate_env" => "NETBOX_AUTO_POPULATE" + "autopopulate_inherit_prefix_length_env" => "NETBOX_AUTO_POPULATE_INHERIT_PREFIX_LENGTH" "autopopulate_subnets_env" => "NETBOX_AUTO_POPULATE_SUBNETS" "default_manuf_env" => "NETBOX_DEFAULT_MANUFACTURER" "default_dtype_env" => "NETBOX_DEFAULT_DEVICE_TYPE" diff --git a/logstash/ruby/netbox_enrich.rb b/logstash/ruby/netbox_enrich.rb index d6b28e6ba..793d4117c 100644 --- a/logstash/ruby/netbox_enrich.rb +++ b/logstash/ruby/netbox_enrich.rb @@ -30,6 +30,13 @@ def concurrency $global_autopopulate_subnets_config_site_hash = Concurrent::Map.new $global_autopopulate_subnets_config_site_hash_populated = Concurrent::AtomicFixnum.new(0) +# Bound lock storage while serializing first observations of the same site/host. +# This prevents two filter instances in this process from creating the same host. +$netbox_identity_locks = Array.new(64) { Mutex.new } +$netbox_mac_write_lock = Mutex.new + +class NetboxIdentityError < StandardError; end + def get_register_cache( cache_type, cache_size, @@ -365,10 +372,21 @@ def register( end @autopopulate = [1, true, '1', 'true', 't', 'on', 'enabled'].include?(_autopopulate_str.to_s.downcase) + # Optionally use a verified local NetBox prefix length for new host IPs. + # This is inventory policy, not a subnet mask measured from packet headers. + _inherit_prefix_str = params["autopopulate_inherit_prefix_length"] + _inherit_prefix_env = params["autopopulate_inherit_prefix_length_env"] + if _inherit_prefix_str.nil? && !_inherit_prefix_env.nil? + _inherit_prefix_str = ENV[_inherit_prefix_env] + end + @autopopulate_inherit_prefix_length = [1, true, '1', 'true', 't', 'on', 'enabled'].include?(_inherit_prefix_str.to_s.downcase) + # fields for device autopopulation @source_hostname = params["source_hostname"] @source_oui = params["source_oui"] @source_mac = params["source_mac"] + @source_vlan = params.fetch("source_vlan", "[network][vlan][id]") + @identity_context_hash = get_register_ttl_cache(:identity_context_hash, 4096, @cache_ttl, true) @source_segment = params["source_segment"] @default_status = params.fetch("default_status", "active").to_sym @@ -651,7 +669,11 @@ def filter( _key.each do |ip_key| - _lookup_tracking_result = getset_with_tracking(_lookup_hash, ip_key) do + # A local MAC match must not leak to a routed or different-VLAN observation + # of the same IP through the shared IP-only cache. + _identity_key = (@lookup_type == :ip_device) ? + [ip_key, event.get("#{@source_mac}"), event.get("#{@source_vlan}"), event.get("#{@source_hostname}")].to_json : ip_key + _lookup_tracking_result = getset_with_tracking(_lookup_hash, _identity_key) do netbox_lookup(event: event, ip_key: ip_key, site_id: _lookup_site_id) end if _lookup_tracking_result[:result] @@ -716,7 +738,7 @@ def filter( result: JSON.generate(_updated_result) }) if @debug end if _updated_result - _lookup_hash[ip_key] = _updated_result + _lookup_hash[_identity_key] = [_updated_result, _key_ip, _nb_queried] _result = deep_copy(_updated_result) end end @@ -1234,6 +1256,155 @@ def lookup_or_create_role( end end +# Identity decisions must distinguish an empty result from an unavailable API. +def identity_records(path, query, nb) + records = [] + query = query.merge(:offset => 0, :limit => @page_size) + loop do + response = nb.get(path, query) + body = response.body + unless response.success? && body.is_a?(Hash) && body[:results].is_a?(Array) + raise NetboxIdentityError, "identity lookup failed: #{path}" + end + page = body[:results] + records.concat(page) + break if page.size < @page_size + query[:offset] += page.size + end + records +end + +def identity_ip_records(ip_str, nb) + # NetBox's address filter matches a host independently of its stored mask. + # Verify that property before allowing any write, even if an API ignores a filter. + host = IPAddr.new(ip_str.split('/').first) + identity_records('ipam/ip-addresses/', { :address => host.to_s }, nb).select do |record| + IPAddr.new(record[:address].to_s.split('/').first) == host + end +end + +def identity_mac(value) + values = Array(value).compact.map { |v| v.to_s.upcase }.uniq + return nil unless values.size == 1 && values.first.match?(/\A[0-9A-F]{2}(?::[0-9A-F]{2}){5}\z/) + mac = values.first + return nil if mac == '00:00:00:00:00:00' || (mac[0, 2].to_i(16) & 1) != 0 + mac +end + +def identity_scope_site(record, nb) + case record[:scope_type] + when 'dcim.site' + record[:scope_id] || record.dig(:scope, :id) + when 'dcim.location' + location = nb.get("dcim/locations/#{record[:scope_id] || record.dig(:scope, :id)}/").body + location.is_a?(Hash) ? location.dig(:site, :id) : nil + when nil + record.dig(:site, :id) # NetBox versions with the older site field + end +end + +def identity_scoped_prefix(ip_str, site_id, nb) + lookup = proc do + host = IPAddr.new(ip_str.split('/').first) + prefixes = identity_records('ipam/prefixes/', { :contains => host.to_s }, nb).select do |prefix| + IPAddr.new(prefix[:prefix].to_s).include?(host) + end + unless prefixes.empty? + longest = prefixes.map { |p| IPAddr.new(p[:prefix]).prefix }.max + prefixes.select! { |p| IPAddr.new(p[:prefix]).prefix == longest } + end + # No VRF is supplied by this filter. Never guess among overlapping contexts. + if prefixes.size == 1 && prefixes.first[:vrf].nil? && prefixes.first.dig(:status, :value) != 'container' + prefix = prefixes.first + prefix_site = identity_scope_site(prefix, nb) + vlan = prefix[:vlan] + vlan_site = nil + if prefix_site.nil? && prefix[:scope_type].nil? && vlan.is_a?(Hash) + vlan = nb.get("ipam/vlans/#{vlan[:id]}/").body + if vlan.is_a?(Hash) + vlan_site = identity_scope_site(vlan, nb) + if vlan_site.nil? && vlan[:scope_type].nil? && vlan.dig(:group, :id) + group = nb.get("ipam/vlan-groups/#{vlan[:group][:id]}/").body + vlan_site = identity_scope_site(group, nb) if group.is_a?(Hash) + end + end + end + if ((prefix_site || vlan_site).to_i == site_id.to_i) + { :cidr => prefix[:prefix], :vlan => vlan.is_a?(Hash) ? vlan[:vid] : nil } + end + end + end + @identity_context_hash ? @identity_context_hash.getset([site_id, ip_str], &lookup) : lookup.call +end + +def identity_observed_prefix(event, ip_str, site_id, nb) + vlans = Array(event.get("#{@source_vlan}")).compact.map(&:to_s).uniq + return nil unless vlans.size == 1 && vlans.first.match?(/\A\d+\z/) + prefix = identity_scoped_prefix(ip_str, site_id, nb) + prefix && prefix[:vlan] && prefix[:vlan].to_i == vlans.first.to_i ? prefix[:cidr] : nil +end + +def identity_local_mac(event, ip_str, site_id, nb) + mac = identity_mac(event.get("#{@source_mac}")) + mac && identity_observed_prefix(event, ip_str, site_id, nb) ? mac : nil +end + +def identity_creation_address(_event, ip_str, site_id, nb) + return ip_str unless @autopopulate_inherit_prefix_length && !ip_str.include?('/') + # The capture VLAN describes the observation link, not a routed endpoint's + # configured network. IP mask policy comes from scoped IPAM; MAC ownership + # still requires the separate observed-VLAN check above. + prefix = identity_scoped_prefix(ip_str, site_id, nb) + prefix ? "#{ip_str}/#{IPAddr.new(prefix[:cidr]).prefix}" : ip_str +end + +def identity_mac_records(mac, nb) + return [] unless mac + identity_records('dcim/mac-addresses/', { :mac_address => mac }, nb).select do |record| + identity_mac(record[:mac_address]) == mac + end +end + +def identity_autopopulated?(device) + Array(device[:tags]).any? { |tag| tag[:slug] == @device_tag_autopopulated[:slug] } +end + +def identity_hostname_candidates(hostname, site_id, nb) + name = hostname.to_s.downcase.delete_suffix('.').split('.').first + return [] if name.to_s.empty? || hostname.to_s.end_with?('.in-addr.arpa') + # Names only prevent another candidate from being created. They never establish + # identity or promote an address found in an interface description. + identity_records('dcim/devices/', { :site_id => site_id, :q => name }, nb).select do |device| + device.dig(:site, :id).to_i == site_id.to_i && + device[:name].to_s.downcase.delete_suffix('.').split('.').first == name + end +end + +def ensure_interface_mac(interface, object_type, mac, nb) + return unless mac && interface.is_a?(Hash) && interface[:id] + $netbox_mac_write_lock.synchronize do + records = identity_mac_records(mac, nb) + record = nil + if records.empty? + response = nb.post('dcim/mac-addresses/', { + :mac_address => mac, :assigned_object_type => object_type, + :assigned_object_id => interface[:id] + }.to_json, @nb_headers) + record = response.body if response.success? && response.body.is_a?(Hash) + elsif records.size == 1 && records.first[:assigned_object_type] == object_type && + records.first[:assigned_object_id] == interface[:id] + record = records.first + end + # Do not claim an unassigned imported MAC, move another interface's MAC, + # or overwrite an existing primary. The caller only supplies auto-created interfaces. + if record && record[:id] && interface[:primary_mac_address].nil? + endpoint = object_type == 'dcim.interface' ? 'dcim/interfaces' : 'virtualization/interfaces' + nb.patch("#{endpoint}/#{interface[:id]}/", { :primary_mac_address => record[:id] }.to_json, @nb_headers) + end + record + end +end + ############################################################################################## # Convert a NetBox-supplied absolute object URL (e.g. the :url field on a device or virtual # machine) into a connection-relative API path that cannot escape the configured NetBox origin. @@ -1304,7 +1475,8 @@ def lookup_devices( lookup_service_port, url_base, url_suffix, - nb + nb, + addresses: nil ) _devices = Array.new _query = { :address => ip_str, @@ -1313,7 +1485,7 @@ def lookup_devices( begin while true do # query all matching IP addresses, but only return devices where the site matches - if (_ip_addresses_response = nb.get('ipam/ip-addresses/', _query).body) && + if (_ip_addresses_response = addresses ? { :results => addresses } : nb.get('ipam/ip-addresses/', _query).body) && _ip_addresses_response.is_a?(Hash) then _tmp_ip_addresses = _ip_addresses_response.fetch(:results, []) @@ -1385,7 +1557,7 @@ def lookup_devices( end end _query[:offset] += _tmp_ip_addresses.size - break unless (_tmp_ip_addresses.size >= @page_size) + break if addresses || (_tmp_ip_addresses.size < @page_size) else # weird/bad response, bail break @@ -1574,14 +1746,14 @@ def create_device_interface( _interface_data = { autopopulate_manuf[:vm] ? :virtual_machine : :device => _autopopulate_device[:id], :name => "e0", :type => "other" } - if !autopopulate_mac.nil? && !autopopulate_mac.empty? - _interface_data[:mac_address] = autopopulate_mac.is_a?(Array) ? autopopulate_mac.first : autopopulate_mac - end if (_interface_create_reponse = nb.post(autopopulate_manuf[:vm] ? 'virtualization/interfaces/' : 'dcim/interfaces/', _interface_data.to_json, @nb_headers).body) && _interface_create_reponse.is_a?(Hash) && _interface_create_reponse.has_key?(:id) then _autopopulate_interface = _interface_create_reponse + ensure_interface_mac(_autopopulate_interface, + autopopulate_manuf[:vm] ? 'virtualization.vminterface' : 'dcim.interface', + identity_mac(autopopulate_mac), nb) elsif @debug puts('create_device_interface (%{name}): _interface_create_reponse: %{result}' % { name: ip_str, result: JSON.generate(_interface_create_reponse) }) end @@ -1627,6 +1799,25 @@ def netbox_lookup( ip_key:, site_id:, previous_result: nil +) + if @lookup_type == :ip_device + lock = $netbox_identity_locks[[site_id, ip_key].hash % $netbox_identity_locks.size] + lock.synchronize do + netbox_lookup_unlocked(event: event, ip_key: ip_key, site_id: site_id, previous_result: previous_result) + end + else + netbox_lookup_unlocked(event: event, ip_key: ip_key, site_id: site_id, previous_result: previous_result) + end +rescue NetboxIdentityError, Faraday::Error, IPAddr::InvalidAddressError => e + puts "netbox identity lookup skipped (#{ip_key}, #{site_id}): #{e.class}" if @debug + return nil, (IPAddr.new(ip_key) rescue nil), false +end + +def netbox_lookup_unlocked( + event:, + ip_key:, + site_id:, + previous_result: nil ) _lookup_result = nil _nb = nil @@ -1644,13 +1835,16 @@ def netbox_lookup( _autopopulate_default_dtype = (@default_dtype.nil? || @default_dtype.empty?) ? "Unspecified" : @default_dtype _autopopulate_hostname = event.get("#{@source_hostname}") _autopopulate_hostname = nil if _autopopulate_hostname.to_s.end_with?('.in-addr.arpa') - _autopopulate_mac = event.get("#{@source_mac}") - _autopopulate_oui = event.get("#{@source_oui}") + # Ethernet addresses on a routed capture belong to the next hop, not the IP + # endpoint. Only use MAC/OUI after a site-scoped prefix/VLAN check succeeds. + _autopopulate_mac = (@lookup_type == :ip_device) ? identity_local_mac(event, ip_key, site_id, _nb) : nil + _autopopulate_oui = _autopopulate_mac ? mac_to_oui_lookup(_autopopulate_mac) : nil _autopopulate_device = nil _autopopulate_role = nil _autopopulate_dtype = nil _autopopulate_manuf = nil + _autopopulate_creation_address = ip_key _prefixes = nil _devices = nil _interface_created = false @@ -1666,11 +1860,78 @@ def netbox_lookup( # then, for those IP addresses, search for devices pertaining to the interfaces assigned to each # IP address (e.g., ipam.ip_address -> dcim.interface -> dcim.device, or # ipam.ip_address -> virtualization.interface -> virtualization.virtual_machine) - _devices = lookup_devices(ip_key, site_id, _lookup_service_port, @netbox_url_base, @netbox_uri_suffix, _nb) + _host_addresses = identity_ip_records(ip_key, _nb) + _devices = lookup_devices(ip_key, site_id, _lookup_service_port, @netbox_url_base, @netbox_uri_suffix, _nb, + addresses: _host_addresses) + if _host_addresses.any? { |a| !a[:vrf].nil? } + _devices = [{ :identity_status => 'vrf_context_required' }] + end + _mac_records = identity_mac_records(_autopopulate_mac, _nb) + _native_mac_devices = [] + if _mac_records.size == 1 && _host_addresses.all? { |a| a[:vrf].nil? } + _mac_record = _mac_records.first + _interface_endpoint = { 'dcim.interface' => 'dcim/interfaces', + 'virtualization.vminterface' => 'virtualization/interfaces' }[_mac_record[:assigned_object_type]] + if _interface_endpoint && _mac_record[:assigned_object_id] + _interface_response = _nb.get("#{_interface_endpoint}/#{_mac_record[:assigned_object_id]}/") + _mac_interface = _interface_response.body + if _interface_response.success? && _mac_interface.is_a?(Hash) && _mac_interface[:vrf].nil? + _native_mac_devices = lookup_devices(ip_key, site_id, _lookup_service_port, @netbox_url_base, @netbox_uri_suffix, _nb, + addresses: [{ :assigned_object => _mac_interface }]).reject { |d| identity_autopopulated?(d) } + end + end + end + _known_ip_devices = _devices.reject { |d| identity_autopopulated?(d) } + if _native_mac_devices.size == 1 && (_known_ip_devices.empty? || + _known_ip_devices.map { |d| d[:url] }.uniq == _native_mac_devices.map { |d| d[:url] }.uniq) + _devices = _native_mac_devices + _devices.each do |device| + device[:identity_source] = 'local_mac' + device[:ip_assignment_conflict] = true unless _host_addresses.any? do |a| + kind = device[:url].to_s.include?('/virtual-machines/') ? :virtual_machine : :device + a.dig(:assigned_object, kind, :id) == device[:id] + end + end + elsif !_native_mac_devices.empty? || _mac_records.size > 1 + _devices = [{ :identity_status => 'conflict_review' }] + end + _devices.each { |d| d[:ip_record_conflict] = true } if _host_addresses.size > 1 + + # A matching IP object without an interface is still an existing host. + # Do not create another /32 or guess which device should own that /24. + if _devices.empty? && !_host_addresses.empty? + _devices = [{ :identity_status => 'unassigned_or_out_of_scope_ip_review' }] + end + if _devices.empty? && !_mac_records.empty? + _devices = [{ :identity_status => 'existing_mac_review' }] + end + if @autopopulate && _devices.empty? + _name_candidates = identity_hostname_candidates(_autopopulate_hostname, site_id, _nb) + unless _name_candidates.empty? + _devices = [{ :identity_status => 'hostname_review', + :candidate_name => _name_candidates.map { |d| d[:name] }, + :candidate_id => _name_candidates.map { |d| d[:id] } }] + end + end + + # Backfill only an unambiguous auto-created interface in the local VLAN. + # Imported interfaces and their existing primary MACs are never changed. + if @autopopulate && _autopopulate_mac && _host_addresses.size == 1 && + _host_addresses.first[:vrf].nil? && _devices.size == 1 && identity_autopopulated?(_devices.first) + _address = _host_addresses.first + _object_type = _address[:assigned_object_type] + _endpoint = { 'dcim.interface' => 'dcim/interfaces', + 'virtualization.vminterface' => 'virtualization/interfaces' }[_object_type] + if _endpoint && _address[:assigned_object_id] + _response = _nb.get("#{_endpoint}/#{_address[:assigned_object_id]}/") + ensure_interface_mac(_response.body, _object_type, _autopopulate_mac, _nb) if _response.success? + end + end if @autopopulate && (_devices.nil? || _devices.empty?) # no results found, autopopulate enabled, eligible IP address... # let's create an entry for this device + _autopopulate_creation_address = identity_creation_address(event, ip_key, site_id, _nb) _autopopulate_device, _autopopulate_role, _autopopulate_dtype, @@ -1787,6 +2048,7 @@ def netbox_lookup( if _device_to_vm # you can't "convert" a device to a VM, so we have to create a new VM then delete the old device + _autopopulate_creation_address = identity_creation_address(event, ip_key, site_id, _nb) _vm_data = { :name => _patched_device_data.fetch(:name, [previous_result.fetch(:name, nil)])&.flatten&.uniq.first, :site => site_id, :tags => _tags, @@ -1814,7 +2076,7 @@ def netbox_lookup( end # create interface+IP for the new VM now so lookup_devices can find it below - _autopopulate_device = create_device_interface(ip_key, + _autopopulate_device = create_device_interface(_autopopulate_creation_address, _autopopulate_device, _autopopulate_manuf, _autopopulate_mac, @@ -1910,7 +2172,7 @@ def netbox_lookup( if !_autopopulate_device.nil? && _autopopulate_device.fetch(:id, nil)&.nonzero? && !_interface_created # device has been created (fresh autopopulate, not a device->VM conversion), we need to create an interface for it - _autopopulate_device = create_device_interface(ip_key, + _autopopulate_device = create_device_interface(_autopopulate_creation_address, _autopopulate_device, _autopopulate_manuf, _autopopulate_mac, @@ -1925,4 +2187,4 @@ def netbox_lookup( ############################################################################### # tests -############################################################################### \ No newline at end of file +############################################################################### diff --git a/shared/tests/fixtures/netbox_identity_harness.rb b/shared/tests/fixtures/netbox_identity_harness.rb new file mode 100644 index 000000000..cac60c43a --- /dev/null +++ b/shared/tests/fixtures/netbox_identity_harness.rb @@ -0,0 +1,420 @@ +# Run the production filter with a deterministic, in-memory NetBox API. +# Optional gems are stubbed only at their loading boundary; the identity, +# creation, pagination, filter cache and synchronization methods run unchanged. +require 'json' +require 'ipaddr' +require 'thread' +%w[faraday fuzzystringmatch lru_reredux stringex_lite].each { |name| $LOADED_FEATURES << "#{name}.rb" } +module Faraday + class Error < StandardError; end +end +module Concurrent + Map = Hash + class AtomicFixnum + def initialize(*); end + end +end +class FakeCache < Hash + def initialize(*); super(); end + def getset(key) + self[key] = yield unless key?(key) + self[key] + end +end +module LruReredux + ThreadSafeCache = FakeCache + module TTL + ThreadSafeCache = FakeCache + end +end +class IdentityFilter + class_eval(File.read(ARGV.fetch(0)), ARGV.fetch(0)) + def initialize + { :netbox_enabled => true, :autopopulate => true, :lookup_type => :ip_device, + :target => '[source][device]', :source => '[source][ip]', :source_mac => '[source][mac]', + :source_vlan => '[network][vlan][id]', :source_hostname => '[source][hostname]', + :source_oui => '[source][oui]', :netbox_url_base => 'http://example.test', + :netbox_uri_suffix => '/api', :page_size => 100, :default_status => :active, + :nb_headers => {}, :cache_size => 100, :cache_ttl => 30, :lookup_cache_size => 20, + :site_lookup_types_hash => FakeCache.new, :identity_context_hash => FakeCache.new, + :device_tag_autopopulated => { :slug => 'malcolm-autopopulated' }, + :device_tag_hostname_unknown => { :slug => 'hostname-unknown' }, + :device_tag_manufacturer_unknown => { :slug => 'manufacturer-unknown' } + }.each { |key,value| instance_variable_set("@#{key}", value) } + end + def lookup_or_create_site(*); { :id => 1, :name => 'Test Site' }; end + def lookup_prefixes(*); []; end + def lookup_or_create_role(*); { :id => 1, :name => 'Test Role' }; end + def lookup_or_create_manuf_and_dtype(*args) + $test_nb.oui_inputs << args.first + [{ :id => 1, :name => 'Test Type' }, { :id => 1, :name => 'Test Vendor', :vm => false }] + end + def mac_to_oui_lookup(*); 'Test Vendor'; end +end +class IdentityFilter::NetBoxConnLazy + def self.new(*); $test_nb; end +end +FakeEvent = Struct.new(:fields) do + def get(key); fields[key]; end + def set(key,value); fields[key] = value; end +end +Response = Struct.new(:body, :status) do + def success?; status >= 200 && status < 300; end +end +class FakeNetbox + attr_accessor :ips, :macs, :prefixes, :fail_path, :devices, :interfaces + attr_reader :writes, :oui_inputs + def initialize + @writes, @oui_inputs = [], [] + native = { :id => 10, :name => 'PC-EXAMPLE', :site => { :id => 1, :name => 'Test Site' }, + :url => 'http://example.test/api/dcim/devices/10/', :tags => [] } + auto = { :id => 20, :name => 'pc-example.local', :site => { :id => 1, :name => 'Test Site' }, + :url => 'http://example.test/api/dcim/devices/20/', :tags => [{ :slug => 'malcolm-autopopulated' }] } + @devices = { 10 => native, 20 => auto } + @interfaces = { 101 => { :id => 101, :device => native, :vrf => nil }, + 201 => { :id => 201, :device => auto, :vrf => nil } } + @ips = [{ :id => 401, :address => '10.10.0.7/32', :vrf => nil, + :assigned_object_type => 'dcim.interface', :assigned_object_id => 201, + :assigned_object => @interfaces[201] }] + @macs = [{ :id => 301, :mac_address => '02:11:22:33:44:55', + :assigned_object_type => 'dcim.interface', :assigned_object_id => 101 }] + @prefixes = [{ :id => 1, :prefix => '10.10.0.0/24', :vrf => nil, + :scope_type => 'dcim.site', :scope_id => 1, :vlan => { :id => 1, :vid => 10 } }] + end + def initialized?; true; end + def response(body,status=200); Response.new(Marshal.load(Marshal.dump(body)),status); end + def get(path,query={}) + return response({},503) if path == @fail_path + data = case path + when 'ipam/ip-addresses/' + @ips.select { |i| IPAddr.new(i[:address].split('/').first).to_s == query[:address] } + when 'dcim/mac-addresses/' + @macs.select { |m| m[:mac_address].upcase == query[:mac_address] } + when 'ipam/prefixes/'; @prefixes + when 'dcim/devices/' + @devices.values.select { |d| d[:name].downcase.include?(query[:q].to_s.downcase) } + when %r{\Adcim/devices/(\d+)/\z}; return response(@devices.fetch($1.to_i)) + when %r{\A(?:dcim|virtualization)/interfaces/(\d+)/\z}; return response(@interfaces.fetch($1.to_i)) + when 'ipam/vlans/1/'; return response({ :id => 1, :vid => 10, :group => { :id => 1 } }) + when 'ipam/vlan-groups/1/'; return response({ :scope_type => 'dcim.site', :scope_id => 1 }) + when 'dcim/locations/1/'; return response({ :site => { :id => 1 } }) + else; raise "unexpected GET #{path} #{query}" + end + response({ :results => data.slice(query.fetch(:offset,0),query.fetch(:limit,100)) || [] }) + end + def post(path,text,*) + data=JSON.parse(text,:symbolize_names => true) + @writes << [:post,path,data] + case path + when 'dcim/devices/' + data[:id] = 900 + data[:site] = { :id => data[:site], :name => 'Test Site' } + data[:role] = { :id => data[:role], :name => 'Test Role' } + data[:device_type] = { :id => data[:device_type], :name => 'Test Type', + :manufacturer => { :id => 1, :name => 'Test Vendor' } } + data[:url] = 'http://example.test/api/dcim/devices/900/' + @devices[900] = data + when 'dcim/interfaces/' + data[:id] = 901 + data[:device] = @devices.fetch(data[:device]) + @interfaces[901] = data + when 'dcim/mac-addresses/' + data[:id] = 902 + @macs << data + when 'ipam/ip-addresses/' + data[:id] = 903 + data[:assigned_object] = @interfaces.fetch(data[:assigned_object_id]) + @ips << data + else; raise "unexpected POST #{path}" + end + response(data,201) + end + def patch(path,text,*) + data=JSON.parse(text,:symbolize_names => true) + @writes << [:patch,path,data] + case path + when %r{\A(?:dcim|virtualization)/interfaces/(\d+)/\z}; @interfaces.fetch($1.to_i).merge!(data) + when %r{\Adcim/devices/(\d+)/\z}; @devices.fetch($1.to_i).merge!(data) + else; raise "unexpected PATCH #{path}" + end + response(data.merge(:id => path.split('/')[-1].to_i)) + end +end + +def event(mac='02:11:22:33:44:55',vlan=10,hostname=nil) + FakeEvent.new({ '[source][ip]' => '10.10.0.7', '[source][mac]' => mac, + '[network][vlan][id]' => vlan, '[source][hostname]' => hostname }) +end +def lookup(filter,event=event()) + filter.netbox_lookup(:event => event, :ip_key => '10.10.0.7', :site_id => 1).first +end +results={} +def check(results,name) + $test_nb = FakeNetbox.new + filter=IdentityFilter.new + raise "failed: #{name}" unless yield(filter,$test_nb) + results[name]=true +end +check(results,'native_mac_replaces_auto_enrichment_without_writes') do |f,nb| + r=lookup(f) + r[:id]==[10] && r[:identity_source]==['local_mac'] && r[:ip_assignment_conflict]==[true] && nb.writes.empty? +end +check(results,'native_mac_without_ip_assignment') do |f,nb| + nb.ips=[] + lookup(f)[:id]==[10] && nb.writes.empty? +end +check(results,'assigned_host_with_24_mask') do |f,nb| + nb.ips.first[:address]='10.10.0.7/24' + nb.ips.first[:assigned_object]=nb.interfaces[101] + nb.ips.first[:assigned_object_id]=101 + lookup(f)[:id]==[10] && nb.writes.empty? +end +check(results,'unassigned_24_does_not_create_32') do |f,nb| + nb.macs=[] + nb.ips=[{ :id => 401, :address => '10.10.0.7/24', :assigned_object_id => nil }] + r=lookup(f) + r[:identity_status]==['unassigned_or_out_of_scope_ip_review'] && nb.writes.empty? && nb.ips.size==1 +end +check(results,'duplicate_masks_flagged_without_writes') do |f,nb| + nb.macs=[] + nb.ips << { :id => 402, :address => '10.10.0.7/24' } + lookup(f)[:ip_record_conflict]==[true] && nb.writes.empty? +end +check(results,'routed_vlan_does_not_assign_mac') do |f,nb| + nb.macs=[] + lookup(f,event('02:11:22:33:44:55',99))[:id]==[20] && nb.writes.empty? +end +check(results,'missing_vlan_does_not_assign_mac') do |f,nb| + nb.macs=[] + lookup(f,event('02:11:22:33:44:55',nil))[:id]==[20] && nb.writes.empty? +end +check(results,'multiple_vlans_do_not_assign_mac') do |f,nb| + nb.macs=[] + lookup(f,event('02:11:22:33:44:55',[10,20]))[:id]==[20] && nb.writes.empty? +end +check(results,'vrf_requires_context') do |f,nb| + nb.ips.first[:vrf]={ :id => 2 } + lookup(f)[:identity_status]==['vrf_context_required'] && nb.writes.empty? +end +check(results,'native_ip_mac_conflict_is_not_overwritten') do |f,nb| + nb.devices[20][:tags]=[] + lookup(f)[:identity_status]==['conflict_review'] && nb.writes.empty? +end +check(results,'duplicate_mac_objects_require_review') do |f,nb| + nb.macs << nb.macs.first.merge(:id => 302) + lookup(f)[:identity_status]==['conflict_review'] && nb.writes.empty? +end +check(results,'hostname_only_prevents_duplicate_without_assignment') do |f,nb| + nb.ips=[] + nb.macs=[] + r=lookup(f,event(nil,nil,'PC-EXAMPLE.local')) + r[:identity_status]==['hostname_review'] && !r.key?(:id) && nb.writes.empty? +end +check(results,'modern_mac_creation_and_repeat_are_idempotent') do |f,nb| + nb.ips=[]; nb.macs=[]; nb.devices={}; nb.interfaces={} + lookup(f) + lookup(f) + posts=nb.writes.select { |w| w[0]==:post } + posts.map { |w| w[1] }.sort==['dcim/devices/','dcim/interfaces/','dcim/mac-addresses/','ipam/ip-addresses/'].sort && + !posts.find { |w| w[1]=='dcim/interfaces/' }[2].key?(:mac_address) && + nb.macs.first[:assigned_object_id]==901 && nb.interfaces[901][:primary_mac_address]==902 && + nb.ips.first[:address]=='10.10.0.7/32' +end +check(results,'backfill_only_auto_interface') do |f,nb| + nb.macs=[] + lookup(f) + nb.macs.size==1 && nb.macs.first[:assigned_object_id]==201 && nb.interfaces[201][:primary_mac_address]==902 +end +check(results,'imported_primary_mac_is_not_modified') do |f,nb| + lookup(f) + nb.interfaces[101][:primary_mac_address].nil? && nb.writes.empty? +end +check(results,'unassigned_existing_mac_is_not_claimed') do |f,nb| + nb.macs.first[:assigned_object_id]=nil + nb.ips=[] + lookup(f)[:identity_status]==['existing_mac_review'] && nb.writes.empty? +end +check(results,'other_interface_mac_is_not_moved') do |f,nb| + f.ensure_interface_mac(nb.interfaces[201],'dcim.interface','02:11:22:33:44:55',nb) + nb.writes.empty? +end +check(results,'existing_primary_is_preserved') do |f,nb| + nb.macs=[]; nb.interfaces[201][:primary_mac_address]={ :id => 999 } + f.ensure_interface_mac(nb.interfaces[201],'dcim.interface','02:11:22:33:44:55',nb) + nb.macs.size==1 && nb.writes.none? { |w| w[0]==:patch } +end +check(results,'vm_mac_assignment_uses_vm_interface_type') do |f,nb| + nb.macs=[] + f.ensure_interface_mac(nb.interfaces[201],'virtualization.vminterface','02:11:22:33:44:55',nb) + nb.macs.first[:assigned_object_type]=='virtualization.vminterface' && + nb.writes.last[1]=='virtualization/interfaces/201/' +end +check(results,'api_failure_does_not_create_inventory') do |f,nb| + nb.fail_path='ipam/ip-addresses/' + lookup(f).nil? && nb.writes.empty? +end +check(results,'pagination_checks_all_mac_objects') do |f,nb| + f.instance_variable_set(:@page_size,1) + nb.macs << nb.macs.first.merge(:id => 302) + lookup(f)[:identity_status]==['conflict_review'] && nb.writes.empty? +end +check(results,'vlan_group_supplies_site_for_unscoped_prefix') do |f,nb| + nb.prefixes.first[:scope_type]=nil; nb.prefixes.first.delete(:scope_id) + lookup(f)[:id]==[10] && nb.writes.empty? +end +check(results,'location_scope_is_resolved') do |f,nb| + nb.prefixes.first[:scope_type]='dcim.location' + lookup(f)[:id]==[10] && nb.writes.empty? +end +check(results,'wrong_site_does_not_trust_mac') do |f,nb| + nb.prefixes.first[:scope_id]=2 + lookup(f)[:id]==[20] && nb.writes.empty? +end +check(results,'ambiguous_prefix_does_not_trust_mac') do |f,nb| + nb.prefixes << nb.prefixes.first.merge(:id => 2) + lookup(f)[:id]==[20] && nb.writes.empty? +end +check(results,'invalid_multicast_and_multiple_macs_are_rejected') do |f,nb| + [nil,[],['02:11:22:33:44:55','02:11:22:33:44:66'],'01:11:22:33:44:55','00:00:00:00:00:00','not-a-mac'].all? { |m| f.identity_mac(m).nil? } +end +check(results,'filter_cache_separates_local_and_routed_observations') do |f,nb| + local=event; routed=event('02:11:22:33:44:55',99) + f.filter(local); f.filter(routed) + local.get('[source][device]')[:id]==[10] && routed.get('[source][device]')[:id]==[20] && nb.writes.empty? +end +check(results,'concurrent_first_observations_create_one_host') do |f,nb| + nb.ips=[]; nb.macs=[]; nb.devices={}; nb.interfaces={} + filters=[f,IdentityFilter.new] + threads=filters.map { |filter| Thread.new { lookup(filter) } } + threads.each(&:value) + nb.writes.count { |w| w[0]==:post && w[1]=='dcim/devices/' }==1 && nb.ips.size==1 && nb.macs.size==1 +end +check(results,'filter_cache_preserves_lookup_tuple_after_update') do |f,nb| + nb.macs=[] + nb.devices[20][:tags] << { :slug => 'hostname-unknown' } + first=event(nil,nil,'NEW-PC'); second=event(nil,nil,'NEW-PC') + f.filter(first); f.filter(second) + first.get('[source][device]')[:name]==['NEW-PC'] && second.get('[source][device]')[:name]==['NEW-PC'] && + nb.writes.count { |w| w[0]==:patch && w[1]=='dcim/devices/20/' }==1 +end +def new_host_fixture(filter,nb) + filter.instance_variable_set(:@autopopulate_inherit_prefix_length,true) + nb.ips=[]; nb.macs=[]; nb.devices={}; nb.interfaces={} +end +check(results,'opt_in_inherits_ipv4_prefix_and_repeat_is_idempotent') do |f,nb| + new_host_fixture(f,nb) + lookup(f); lookup(f) + nb.ips.size==1 && nb.ips.first[:address]=='10.10.0.7/24' && + nb.writes.count { |w| w[0]==:post && w[1]=='dcim/devices/' }==1 +end +check(results,'inheritance_uses_most_specific_prefix') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes << nb.prefixes.first.merge(:id=>2,:prefix=>'10.10.0.0/26') + lookup(f) + nb.ips.first[:address]=='10.10.0.7/26' +end +check(results,'inheritance_without_mac_still_checks_vlan') do |f,nb| + new_host_fixture(f,nb) + lookup(f,event(nil,10)) + nb.ips.first[:address]=='10.10.0.7/24' && nb.macs.empty? +end +check(results,'inheritance_default_off_keeps_host_mask') do |f,nb| + new_host_fixture(f,nb) + f.instance_variable_set(:@autopopulate_inherit_prefix_length,false) + lookup(f) + nb.ips.first[:address]=='10.10.0.7/32' +end +check(results,'inheritance_missing_capture_vlan_uses_scoped_prefix') do |f,nb| + new_host_fixture(f,nb) + lookup(f,event(nil,nil)) + nb.ips.first[:address]=='10.10.0.7/24' && nb.macs.empty? +end +check(results,'inheritance_routed_vlan_uses_prefix_without_gateway_mac') do |f,nb| + new_host_fixture(f,nb) + lookup(f,event('02:11:22:33:44:55',99)) + nb.ips.first[:address]=='10.10.0.7/24' && nb.macs.empty? && nb.oui_inputs.all?(&:nil?) +end +check(results,'inheritance_ambiguous_prefix_keeps_host_mask') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes << nb.prefixes.first.merge(:id=>2) + lookup(f,event(nil,10)) + nb.ips.first[:address]=='10.10.0.7/32' +end +check(results,'inheritance_wrong_site_keeps_host_mask') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes.first[:scope_id]=2 + lookup(f,event(nil,10)) + nb.ips.first[:address]=='10.10.0.7/32' +end +check(results,'inheritance_unresolved_vrf_keeps_host_mask') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes.first[:vrf]={ :id=>2 } + lookup(f,event(nil,10)) + nb.ips.first[:address]=='10.10.0.7/32' +end +check(results,'inheritance_missing_prefix_keeps_host_mask') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes=[] + lookup(f,event(nil,10)) + nb.ips.first[:address]=='10.10.0.7/32' +end +check(results,'inheritance_does_not_rewrite_existing_32') do |f,nb| + f.instance_variable_set(:@autopopulate_inherit_prefix_length,true) + lookup(f) + nb.ips.first[:address]=='10.10.0.7/32' && nb.writes.empty? +end +check(results,'inheritance_preserves_explicit_input_mask') do |f,nb| + f.instance_variable_set(:@autopopulate_inherit_prefix_length,true) + f.identity_creation_address(event,'10.10.0.7/27',1,nb)=='10.10.0.7/27' && nb.writes.empty? +end +check(results,'inheritance_supports_ipv6_without_truncating_host') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes.first[:prefix]='fd00:10::/64' + e=event(nil,10); e.set('[source][ip]','fd00:10::7') + f.netbox_lookup(:event=>e,:ip_key=>'fd00:10::7',:site_id=>1) + nb.ips.first[:address]=='fd00:10::7/64' +end +check(results,'inheritance_ipv6_fallback_is_128') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes=[] + e=event(nil,10); e.set('[source][ip]','fd00:10::7') + f.netbox_lookup(:event=>e,:ip_key=>'fd00:10::7',:site_id=>1) + nb.ips.first[:address]=='fd00:10::7/128' +end +check(results,'inheritance_api_failure_does_not_leave_new_device') do |f,nb| + new_host_fixture(f,nb) + nb.fail_path='ipam/prefixes/' + lookup(f,event(nil,10)).nil? && nb.writes.empty? +end +check(results,'inheritance_site_scoped_prefix_without_vlan') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes.first[:vlan]=nil + lookup(f,event('02:11:22:33:44:55',10)) + nb.ips.first[:address]=='10.10.0.7/24' && nb.macs.empty? +end +check(results,'inheritance_rejects_container_prefix') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes.first[:status]={ :value=>'container' } + lookup(f,event(nil,nil)) + nb.ips.first[:address]=='10.10.0.7/32' && nb.macs.empty? +end +check(results,'inheritance_known_hosts_before_routed_conn_creates_one_24') do |f,nb| + new_host_fixture(f,nb) + f.filter(event(nil,nil)) + f.filter(event('02:11:22:33:44:55',99)) + nb.ips.size==1 && nb.ips.first[:address]=='10.10.0.7/24' && nb.macs.empty? && + nb.writes.count { |w| w[0]==:post && w[1]=='dcim/devices/' }==1 +end +check(results,'inheritance_unknown_site_is_not_guessed') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes.first[:scope_type]=nil;nb.prefixes.first.delete(:scope_id);nb.prefixes.first[:vlan]=nil + lookup(f,event(nil,nil)) + nb.ips.first[:address]=='10.10.0.7/32' +end +check(results,'inheritance_site_scoped_vlan_group_without_capture_vlan') do |f,nb| + new_host_fixture(f,nb) + nb.prefixes.first[:scope_type]=nil;nb.prefixes.first.delete(:scope_id) + lookup(f,event(nil,nil)) + nb.ips.first[:address]=='10.10.0.7/24' && nb.macs.empty? +end +puts JSON.generate(results) diff --git a/shared/tests/test_netbox_identity.py b/shared/tests/test_netbox_identity.py new file mode 100644 index 000000000..77765c129 --- /dev/null +++ b/shared/tests/test_netbox_identity.py @@ -0,0 +1,28 @@ +"""Exercise the production Ruby NetBox filter without optional gems or a server.""" + +import json +import shutil +import subprocess +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[2] +HARNESS = Path(__file__).parent / 'fixtures' / 'netbox_identity_harness.rb' +SCRIPT = ROOT / 'logstash' / 'ruby' / 'netbox_enrich.rb' + + +def test_netbox_identity_regressions(): + ruby = shutil.which('ruby') + if ruby is None: + pytest.skip('requires ruby') + result = subprocess.run( + [ruby, str(HARNESS), str(SCRIPT)], + text=True, + capture_output=True, + check=True, + timeout=90, + ) + checks = json.loads(result.stdout) + assert len(checks) == 49 + assert all(checks.values()), checks