The full calibration protocol uses five repetitions of idle and AllReduce at 1, 4, 16, 64, and 128 MiB at one selected supported interval. A bounded idle collector study compares 1.0, 0.5, and 0.2 second polling and records jitter and collector duty/overhead. The capture gate defaults to an idle median plus three MAD with an explicit floor. The accepted sensitivity range is payload-specific; a small-payload miss does not silently become universal support or universal failure.
Feature engineering aggregates complete runs for exported research tables and
preserves communication_only, auxiliary_only, and
communication_plus_auxiliary. The primary binary task is training versus
inference. Controls remain a separate diagnostic class and are never relabeled
as inference. A secondary shallow-tree diagnostic reports training versus
inference versus control with an explicit three-class confusion matrix.
The primary unit is a complete run. Windows from a run never cross splits. Startup is explicitly excluded by default and can be analyzed separately. Standard profiles use at least three repetitions and randomize bounded run order. A second Kaggle session is required before cross-session claims.
Primary splitting uses true experiment_session_id values and follows a fixed
hierarchy. With at least three suitable sessions, whole sessions become train,
validation, and test only when every partition retains both target classes and
all required families. Otherwise, deterministic class/family-stratified whole
runs are used. Each required family needs at least three runs so it can appear
in all three partitions. Missing or ambiguous legacy session identity is a hard
failure; environment fingerprints are never grouping keys. Family and
configuration holdouts are separate diagnostics and may explicitly have a
one-class test partition, in which case balanced metrics are undefined.
The SDK also exposes complete-group GroupKFold and LeaveOneGroupOut planners. They group by run for window-safe cross-validation or by true experiment session for leave-one-session-out diagnostics. No fold may split a run. These folds supplement the predefined train/validation/test protocol and are used only when the available group count and class coverage make metrics meaningful.
Sampling calibration compares 1.0, 0.5, and 0.2 second intervals. The collector records actual intervals, jitter, overruns, field missingness, and a conservative sampler duty fraction. The optional 0.1-second interval remains diagnostic until its overhead and jitter are measured.
Calibration records five idle-baseline observations and five repetitions at
each of 1, 4, 16, 64, and 128 MiB in the full path. Its idle-relative capture
threshold, per-payload 80% capture rate, minimum repetitions, rank correlation,
and dynamic-range criteria are specified in
nvidia-calibration.md. Per-payload summaries also
report mean, median absolute deviation, and coefficient of variation. These
thresholds are pragmatic gates, not universal physical laws. Calibration can be
supported, partially_supported, inconclusive, not_supported, or
failed; partial support applies only to the reported reliable payload groups,
while inconclusive means the evidence cannot decide support. Missing idle data
or one-shot payloads cannot support new evidence. Calibration failure prevents
standard/extended detector collection unless explicit negative-calibration
research mode is selected.
The final corpus manifest and feature-extraction summary bind the exact current- session calibration path, SHA-256, session/collection/environment/source/schema metadata, status, and current-versus-prior-session label. Evaluation verifies that reference and never discovers calibration by filename order. A restored prior-session calibration is provenance input only; the fresh current-session calibration is the collection gate.
Feature extraction requires an explicit versioned corpus manifest and produces one coverage decision for every planned slot. Calibration-designated controls cannot enter a benign selection. Coverage records expose exact per-GPU row counts, common overlap, warmup, usable duration, sampling gaps, aligned pairs, and per-window output counts. A missing or invalid run is never silently skipped.
The primary window is 30 seconds. Five- and 15-second windows are separately labeled diagnostic short-window analyses; they are useful for diagnosing old pilot evidence but never substitute for primary coverage. Primary evaluation requires every required benign family and its configured minimum run count to have at least one 30-second window before any model fitting or metric calculation.
Cross-GPU features use deterministic, no-reuse timestamp pairing within a configurable tolerance. Independently filtered device arrays are not paired by position. Per-GPU TX/RX features likewise use values supported in the same telemetry sample.
Features include distribution summaries, variation, slopes, autocorrelation, idle/duty fractions, PCIe totals/ratios, missingness, cross-GPU divergence, and cross-correlation. Evaluation compares mean-communication and duty-cycle thresholds, logistic regression, a shallow decision tree, and one secondary random forest across communication-only, auxiliary-only, and combined sets. It reports window and run metrics, per-family errors, uncalibrated probability quality diagnostics, and an abstention region.
The primary result is the 30-second communication-only (currently NVML PCIe) block with window/run sample counts. Candidate models are fitted on train only, including median imputation, variance filtering, and logistic scaling. Model and probability-threshold selection use validation only; test data is evaluated once and cannot affect those choices. Probabilities are not claimed to be calibrated. Non-PCIe, combined-feature, short-window, family-holdout, and configuration-holdout results are labeled diagnostics. Host-transfer and synchronized-inference false-positive rates are reported explicitly when those families occur in test.
Run-level uncertainty intervals require at least 20 independent test runs. A
smaller pilot reports an insufficient independent groups warning and leaves
the interval null instead of presenting a misleading narrow estimate.
Benign workloads are duration-controlled. Each declares warmup, minimum steady-state measurement duration, and an optional iteration cap. The current defaults collect at least 35 measured seconds after a five-second warmup; the margin prevents asynchronous one-Hz sampler edges from reducing the common telemetry interval below the 30-second primary window. A supplied cap causes an explicit failure if duration remains unmet. Rank-local interval events are validated and their intersection is persisted in the run manifest.
The standard benign pilot has the eight acceptance-criteria family IDs and three repetitions per family. Its corpus plan is create-only and precedes calibration and workload execution. A distinct final manifest assigns only completed runs, after which the coverage summary reports planned, completed, failed, and primary-feature-valid counts per family. The extended profile is explicitly opt-in and adds bounded batch, sequence, model-size, precision, barrier-cadence, transfer, and optional peer-copy variants. See the benign workload matrix.
Falsification includes unresponsive counters, substantial class overlap, grouped-score collapse, duration/startup/framework leakage, hard-negative false positives, adversarial false negatives, and session instability.
Adversarial evaluation is a later, explicit-approval stage. Its frozen baseline fits benign training rows only; adversarial rows cannot affect preprocessing, model selection, or threshold selection. A sealed plan keeps one family plus declared session/configuration IDs untouched through development and hardening. Training variants report evasion with duration/throughput/loss/memory/sync and communication proxies, while the synthetic non-training decoy reports false positives. See bounded adversarial research.
The method is adapted from the locally supplied literature, especially the
broad telemetry/adversarial evaluation structure of Rahman and Tajdari and the
communication-structuring questions of Seferis and Fist. CommGuard does not
import their empirical results. Full attribution and scope differences are in
references.md.