feat: add NICo hardware ingestion and DPU health validations (#133, #136)

[abegnoche]

Summary

Re-homes the NICo (formerly Carbide/Forge) hardware-lifecycle validations onto main's current conventions (labels not markers, Apache-2.0 SPDX headers, the suites/ + providers/<name>/{config,scripts}/ model). Fresh re-implementation, not a rebase of #228.

• Adds HardwareIngestionCheck, DpuHealthCheck, and DpuNetworkCheck validation classes, labeled bare_metal.
• Folds the hardware-ingestion and DPU-health validations into the canonical suites/bare_metal.yaml contract (no separate per-feature suites or HARDWARE platform).
• Adds the NICo provider under providers/nico/: a single config/bare_metal.yaml that imports the BM suite and implements the verify_ingestion + check_dpu_health steps, a shared NICo API client, and the two scripts. NICo implements only those two steps today, so the rest of the BM suite is skipped automatically (step_not_configured); wiring the full lifecycle is a follow-up.
• Adds NICo readiness checks to isvctl doctor (env/auth/API reachability) and redaction for NICo secrets.
• Documents the NICo steps in the Bare Metal section of configs/suites/README.md.

Behavior notes / decisions

• Ingestion joins expected vs discovered machines by the authoritative expectedMachine.machineId (not chassis serial): missing = unlinked, extra = discovered-but-unexpected.
• The machine id (UUID) is the human-facing identifier in all validation output. chassis_serial is retained only as a debug field (manifest serial for ingestion; dmiData.chassisSerial for DPU health, never falling back to the id).
• Both validations emit a passing subtest per healthy machine and name the offending machine (id + serial + failing value) in the failure summary.
• check_dpu_health filters non-DPU machines client-side so a missing/ignored server-side filter can't cause false failures; nullable count and list fields are handled defensively.

Test plan

• uv run pytest isvtest/tests/test_hardware.py isvctl/tests/test_nico_provider.py -- pass
• make test (all packages) + make lint -- pass
• ruff lint/format + SPDX header check pass
• End-to-end: ISVTEST_INCLUDE_UNRELEASED=1 isvctl test run -f providers/nico/config/bare_metal.yaml -- only ingestion_check + dpu_health run, the rest of the BM suite is skipped (step_not_configured)
• Live run against a NICo site: ingestion correctly flagged an Error-state machine; DPU health passed across 20 machines
• Broader live API validation against additional NICo sites

Notes

• DpuNetworkCheck is unit-tested but not yet wired into a suite (it needs a live instance with EVPN overlay).
• New checks are gated out of released_tests.json until a version bump; run locally with ISVTEST_INCLUDE_UNRELEASED=1.

Closes #133
Closes #136

Summary by CodeRabbit

• New Features
  • NICo provider added for bare-metal validations with two new validation steps: hardware ingestion and DPU health; CLI checks emit JSON reports.
• Doctor
  • doctor now includes NICo readiness and authentication checks (bearer token or OIDC) and probes NICo API.
• Documentation
  • Bare-metal suite README updated to list the new steps.
• Chores
  • NICo auth secrets added to sensitive-env redaction.
• Tests
  • Extensive tests for NICo auth flows, scripts, pagination, and new hardware validations.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds NICo provider support: shared auth/API client, hardware ingestion and DPU health scripts, validation classes, doctor CLI checks, bare-metal config wiring, and tests.

Changes

NICo Provider Integration

Layer / File(s)	Summary
NICo shared client library isvctl/configs/providers/nico/scripts/common/__init__.py, isvctl/configs/providers/nico/scripts/common/nico_client.py	Environment-driven auth (bearer token or OIDC discovery + client_credentials), token acquisition/error wrapping, authenticated GET, paginated retrieval across multiple response shapes, health classification, and capability summation.
Hardware ingestion verification isvctl/configs/providers/nico/scripts/hardware_ingestion/verify_ingestion.py, isvtest/src/isvtest/validations/hardware.py (HardwareIngestionCheck)	Script fetches expected vs actual machines, joins by machineId, emits ingestion counts and per-machine status/health/capabilities. Validation enforces thresholds, detects missing/extra machines, and validates per-machine status/health.
DPU health checking isvctl/configs/providers/nico/scripts/dpu/check_dpu_health.py, isvtest/src/isvtest/validations/hardware.py (DpuHealthCheck, DpuNetworkCheck)	Script filters DPU-capable machines, extracts DPU alerts, successes, heartbeat, and counts; emits per-machine JSON. Validations check DPU presence/counts, heartbeat requirement, health alerts, interface readiness, BGP, and extension deployment statuses.
Doctor CLI NICo provider support isvctl/src/isvctl/doctor/checks/env.py, isvctl/src/isvctl/redaction.py	Adds NICo env checks, OIDC discovery/token logic and API probe, registers NICo strict env vars to avoid duplicate reporting, and redacts NICo secret env vars.
Configuration and wiring isvctl/configs/providers/nico/config/bare_metal.yaml, isvctl/configs/suites/bare_metal.yaml, isvctl/configs/suites/README.md	Adds NICo bare-metal provider config that imports the shared suite, wires verify_ingestion and check_dpu_health steps with templated args and timeouts, and documents the new steps and validation groups.
Comprehensive test coverage isvctl/tests/test_doctor_cli.py, isvctl/tests/test_nico_provider.py, isvtest/tests/test_hardware.py	Tests cover doctor CLI NICo auth flows (bearer and OIDC), OIDC error-body handling, NICo client pagination/response shapes, NICo provider config wiring, DPU health script normalization/filtering, and validation scenarios for ingestion, DPU health, and network checks.

sequenceDiagram
  participant CLI as Validation CLI
  participant ResolveAuth as resolve_auth
  participant Issuer as OIDC Issuer
  participant TokenEP as TokenEndpoint
  participant NICoAPI as NICo API
  participant Validator as isvtest Validations
  CLI->>ResolveAuth: request token (NICO_BEARER_TOKEN or OIDC)
  alt bearer token present
    ResolveAuth-->>CLI: token (NICO_BEARER_TOKEN)
  else OIDC path
    ResolveAuth->>Issuer: GET /.well-known/openid-configuration
    Issuer-->>ResolveAuth: token_endpoint
    ResolveAuth->>TokenEP: POST client_credentials (Basic auth)
    TokenEP-->>ResolveAuth: access_token
    ResolveAuth-->>CLI: token (oidc_client_credentials)
  end
  CLI->>NICoAPI: forge_get_all (Bearer) for machines/expectedMachines
  NICoAPI-->>CLI: paginated JSON
  CLI->>Validator: provide step outputs for HardwareIngestion/DpuHealth/DpuNetwork
  Validator-->>CLI: pass/fail + subtest results

Loading

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main addition: NICo hardware ingestion and DPU health validations, and references the linked issues.
Linked Issues check	✅ Passed	The PR implements all coding requirements from both issues: #133 verifies machine ingestion/matching, #136 validates DPU health status; both are fully implemented via HardwareIngestionCheck, DpuHealthCheck classes and supporting NICo provider infrastructure.
Out of Scope Changes check	✅ Passed	All changes are in-scope: NICo provider setup, validation classes, doctor CLI support, tests, and documentation directly serve the stated objectives of hardware ingestion and DPU health checking.
Docstring Coverage	✅ Passed	Docstring coverage is 86.87% which is sufficient. The required threshold is 80.00%.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/nico-hardware-validation

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

🔐 TruffleHog Secret Scan

✅ No secrets or credentials found!

Your code has been scanned for 700+ types of secrets and credentials. All clear! 🎉

🔗 View scan details

_{🕐 Last updated: 2026-06-05 20:00:02 UTC | Commit: 8aaaafe}

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (1)

isvctl/configs/providers/nico/scripts/hardware_ingestion/verify_ingestion.py (1)

162-162: ⚡ Quick win

Stabilize capabilities ordering for deterministic output.

Line 162 converts a set to a list, so output order can vary run-to-run. Sorting here avoids noisy diffs and brittle output assertions.

Proposed fix

-            cap_types = list({c.get("type", "") for c in capabilities})
+            cap_types = sorted({c.get("type", "") for c in capabilities if c.get("type")})

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@isvctl/configs/providers/nico/scripts/hardware_ingestion/verify_ingestion.py`
at line 162, The conversion of capabilities to cap_types is nondeterministic
because it builds a list from a set; replace list({c.get("type", "") for c in
capabilities}) with a deterministic sorted call—e.g., cap_types =
sorted({c.get("type", "") for c in capabilities})—so that cap_types is stable
across runs; ensure any downstream code expecting list order still works with
the sorted order.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@isvctl/configs/providers/nico/scripts/common/nico_client.py`:
- Around line 242-266: The pagination logic clamps the requested page size to
100 when building page_params but still compares len(items) against the original
page_size, causing early termination when page_size > 100; fix by computing an
effective_page_size = min(page_size, 100), use str(effective_page_size) when
setting page_params["pageSize"] and use effective_page_size in the end-of-loop
check (replace len(items) < page_size with len(items) < effective_page_size) so
paging continues correctly; update references in the function that builds
page_params/handles items (page_params, pageSize, page_number, items,
all_items).

In `@isvctl/configs/providers/nico/scripts/dpu/check_dpu_health.py`:
- Around line 78-101: Tighten the loose dict typings and add a docstring for
main: update helper signatures _lower_field, _is_dpu_alert, _has_dpu_heartbeat,
and _extract_health_successes to use PEP 585 annotations (e.g., dict[str, Any]
for JSON-like inputs and list[str] for string lists) instead of plain dict, and
type any local result variables accordingly; also add a concise PEP 257
docstring to main() describing its purpose, parameters (if any) and return
value. Ensure you import Any from typing if needed and keep existing function
names (_lower_field, _is_dpu_alert, _has_dpu_heartbeat,
_extract_health_successes, main) unchanged so the changes are only to
annotations and docstring.

In
`@isvctl/configs/providers/nico/scripts/hardware_ingestion/verify_ingestion.py`:
- Line 85: Add a PEP 257 docstring to the main() function describing its
purpose, parameters (if any) and return value, and update all broad container
type hints to PEP 585 parameterized forms (e.g., replace dict with dict[str,
Any] or the specific key/value types, replace list[dict] with list[dict[str,
Any]] or list[SpecificType]) so typing is explicit; apply the same change to the
other functions in this module that currently use unparameterized containers
(the ones flagged around the later annotations) and import typing.Any if needed.

In `@isvctl/tests/test_nico_provider.py`:
- Around line 56-63: Add PEP 257 docstrings to the helper functions
_load_nico_client and _load_dpu_health_script describing their purpose,
parameters (if any), and return value, and update the _isolated_common_imports
signature to include an explicit return type (e.g.,
typing.ContextManager[ModuleType] or the appropriate ContextManager[...] type
used in the file); locate these symbols in the test file and ensure the
docstrings are concise and the return type annotation is added to the function
definition.

In `@isvtest/src/isvtest/validations/hardware.py`:
- Line 29: The helper _machine_label currently types its parameter as a plain
dict and several run() methods lack docstrings; update _machine_label signature
to use PEP 585 annotations (e.g., machine: dict[str, Any]) and add the necessary
"from typing import Any" import if missing, and add concise PEP 257-compatible
docstrings to each run() method (describe purpose, parameters, return value) so
all functions and methods follow the repo's typing and docstring standards;
locate symbols named _machine_label and any methods named run to apply these
changes.

---

Nitpick comments:
In
`@isvctl/configs/providers/nico/scripts/hardware_ingestion/verify_ingestion.py`:
- Line 162: The conversion of capabilities to cap_types is nondeterministic
because it builds a list from a set; replace list({c.get("type", "") for c in
capabilities}) with a deterministic sorted call—e.g., cap_types =
sorted({c.get("type", "") for c in capabilities})—so that cap_types is stable
across runs; ensure any downstream code expecting list order still works with
the sorted order.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 980a1860-2290-4b87-9b17-f2f5ac650ed3

📥 Commits

Reviewing files that changed from the base of the PR and between d7d89fe and 8aaaafe.

📒 Files selected for processing (13)

• isvctl/configs/providers/nico/config/bare_metal.yaml
• isvctl/configs/providers/nico/scripts/common/__init__.py
• isvctl/configs/providers/nico/scripts/common/nico_client.py
• isvctl/configs/providers/nico/scripts/dpu/check_dpu_health.py
• isvctl/configs/providers/nico/scripts/hardware_ingestion/verify_ingestion.py
• isvctl/configs/suites/README.md
• isvctl/configs/suites/bare_metal.yaml
• isvctl/src/isvctl/doctor/checks/env.py
• isvctl/src/isvctl/redaction.py
• isvctl/tests/test_doctor_cli.py
• isvctl/tests/test_nico_provider.py
• isvtest/src/isvtest/validations/hardware.py
• isvtest/tests/test_hardware.py