Add Bytes over connection as edge size and color in net flow graph - #126
Merged
Conversation
zasgar
approved these changes
Sep 24, 2020
entlein
added a commit
to k8sstormcenter/pixie
that referenced
this pull request
Aug 7, 2026
…le allowlist) Part A of aeprod28. Adds dx_execve, dx_vfs_events, dx_unlink, dx_dlookup, dx_mprotect, dx_creds, dx_bpf, dx_ptrace — the pid-keyed tracepoint tables for the dark vectors (V1/V2/V6/V7/V8). Generic 4-file allowlist edit (schema.sql, ddl.go KnownTables+PixieTables, apply.go OperatorOwnedTables, pxl/tables.go builtinTables) + the count guard. DDL is one-col/line (line-oriented verify parser); all carry the requiredPixieColumns (namespace/pod/hostname/time_) so VerifyPixieSchema passes. clickhouse + pxl tests green. TODO aeprod28: pid->pod pull enrichment, AE-owned no-TTL tracepoint deploy, pgsql-timeout (#7), steering rework (pixie-io#93/#62). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017k7uYSNUctQvkTAZYJbaB3 Signed-off-by: entlein <einentlein@gmail.com>
entlein
added a commit
to k8sstormcenter/pixie
that referenced
this pull request
Aug 7, 2026
…tables (pixie-io#126) Part B+enrichment for aeprod28. #7 pgsql write: the firehose pull bounded query+write by cfg.Refresh (~30s) — far too tight for pgsql_events (full SQL text + heavy socket_tracer parse), so ExecuteScript hit context-deadline and pgsql landed 0 rows. Added a dedicated QueryTimeout (env-overridable, default 150s, matching the OrderQuery 180s budget); the pull now bounds the query by it, not Refresh. pixie-io#126 pid->pod enrichment: the 8 dark-vector tracepoint tables emit raw kernel pid (no upid) — the native px.upid_to_pod_name path fails on them. Added PodEnrichPxL: native tables keep upid resolution; dark tables merge process_stats on pid ONLY (the validated join-pod.pxl query — NOT pid+asid, since px.asid() is the kelvin asid on a dynamic tracepoint, not the per-PEM asid). Wired into both the passthrough (CompilePassthrough) and targeted (QueryFor) builders; the OrderQuery pod filter uses bare-pod equality for dark tables (their df.pod is the bare name). Test: TestPodEnrichPxL_DarkVsNative. Build + pxl/passthrough/clickhouse tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017k7uYSNUctQvkTAZYJbaB3 Signed-off-by: entlein <einentlein@gmail.com>
entlein
added a commit
to k8sstormcenter/pixie
that referenced
this pull request
Aug 7, 2026
…n (8 dx_* tables) (#89) * feat(ae pixie-io#126): 8 dark-vector tracepoint tables (schema + 4-file allowlist) Part A of aeprod28. Adds dx_execve, dx_vfs_events, dx_unlink, dx_dlookup, dx_mprotect, dx_creds, dx_bpf, dx_ptrace — the pid-keyed tracepoint tables for the dark vectors (V1/V2/V6/V7/V8). Generic 4-file allowlist edit (schema.sql, ddl.go KnownTables+PixieTables, apply.go OperatorOwnedTables, pxl/tables.go builtinTables) + the count guard. DDL is one-col/line (line-oriented verify parser); all carry the requiredPixieColumns (namespace/pod/hostname/time_) so VerifyPixieSchema passes. clickhouse + pxl tests green. TODO aeprod28: pid->pod pull enrichment, AE-owned no-TTL tracepoint deploy, pgsql-timeout (#7), steering rework (pixie-io#93/#62). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017k7uYSNUctQvkTAZYJbaB3 Signed-off-by: entlein <einentlein@gmail.com> * feat(ae): pgsql firehose timeout (#7) + pid->pod enrichment for dark tables (pixie-io#126) Part B+enrichment for aeprod28. #7 pgsql write: the firehose pull bounded query+write by cfg.Refresh (~30s) — far too tight for pgsql_events (full SQL text + heavy socket_tracer parse), so ExecuteScript hit context-deadline and pgsql landed 0 rows. Added a dedicated QueryTimeout (env-overridable, default 150s, matching the OrderQuery 180s budget); the pull now bounds the query by it, not Refresh. pixie-io#126 pid->pod enrichment: the 8 dark-vector tracepoint tables emit raw kernel pid (no upid) — the native px.upid_to_pod_name path fails on them. Added PodEnrichPxL: native tables keep upid resolution; dark tables merge process_stats on pid ONLY (the validated join-pod.pxl query — NOT pid+asid, since px.asid() is the kelvin asid on a dynamic tracepoint, not the per-PEM asid). Wired into both the passthrough (CompilePassthrough) and targeted (QueryFor) builders; the OrderQuery pod filter uses bare-pod equality for dark tables (their df.pod is the bare name). Test: TestPodEnrichPxL_DarkVsNative. Build + pxl/passthrough/clickhouse tests green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017k7uYSNUctQvkTAZYJbaB3 Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: dc_snoop replaces execve in the dark-table export set The custom bprm_execve kprobe does not fire on 6.x kernels (inlined/renamed); the shipped Pixie dc_snoop (kprobe:lookup_fast) is the working process+file instrument and captures live (MANIFESTO §10). Swap the dark-vector export: - tables.go: dx_execve -> dx_dcsnoop in builtinTables. - schema.sql / ddl.go / apply.go: dx_execve DDL -> dx_dcsnoop, + the dc_snoop `t` column (R=reference / M=miss from the dcache lookup). - compile.go: dx_dcsnoop is a pid-keyed dark-vector table (PodEnrichPxL merges process_stats on pid for pod+namespace). One dentry-cache probe serves both R0001 (process launch = binary lookup) and R0002 (file access). 14 AE packages green. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: dark-vector pod filter must use the namespaced pod key proc.ctx['pod'] yields the NAMESPACED pod name (ns/pod) on Pixie v0.14.20+, not the bare pod name the ported comment assumed. Verified live (rig 6a5f6bc0): df.pod=='specimen/activity-gen-xxx' matches 2 rows, df.pod=='activity-gen-xxx' matches 0 — so every dark-vector pull silently returned empty while the native protocol pulls (namespaced key) worked. Match the namespaced key for dark vectors too. Native protocol export already proven end-to-end (dns/conn land in forensic_db for the steered pod). Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: register dc_snoop + stack_trace + creds_change retention scripts at boot The operator already reconciles ClickHouse-plugin retention scripts on boot (installPresetScripts, gated INSTALL_PRESET_SCRIPTS=true: GetClusterScripts → purge operator-managed → AddDataRetentionScript for each builtin). Add three dark-vector/profiler export scripts to the builtin set so they are registered IF-NOT-PRESENT, permanently, via the native OTel→ClickHouse plugin — no external wrapper: - ch-dc_snoop — UpsertTracepoint(lookup_fast, "876000h" ≈ permanent) + px.export (dentry cache = process+file, V1/V2). bare bprm_execve/d_lookup don't fire on 6.x (MANIFESTO §10); lookup_fast does. - ch-stack_trace — native continuous profiler stack_traces.beta (V9, no tracepoint) → px.export. The OTel "profiles" stack-trace signal. - ch-creds_change — UpsertTracepoint(commit_creds, "876000h") + px.export: a process committing new uid==0 while its previous real uid>0 = privilege escalation to root (V7). Scripts embedded (go:embed internal/script/presets/*.pxl); each uses px.plugin.start_time/end_time (plugin-managed window) + px.otel.ClickHouseRows. Tracepoints use a ~100y TTL (no built-in "permanent" flag; a huge TTL is effectively permanent + survives a cron stall). CH DDL for the 3 tables added to schema.sql so the operator self-creates them. 14 AE packages green. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: TrimSpace API key / host / DSN / cluster-id (fixes RST_STREAM PROTOCOL_ERROR) A pixie-api-key sourced via a k8s secret created with `kubectl --from-file` keeps the file's trailing newline. Sent as the pixie-api-key gRPC metadata header to the cloud PluginService, that newline is an HTTP/2 header protocol violation → the server replies with RST_STREAM PROTOCOL_ERROR. It surfaces as "could not ensure ClickHouse plugin is enabled" / "get cluster scripts: ... PROTOCOL_ERROR" and blocks retention-script registration entirely, even though the key is valid (verified: same key via $(...) — which strips the newline — lists the scripts fine). TrimSpace the API key (and defensively the endpoint host, ClickHouse DSN, and cluster id) so a whitespace-padded secret can't break the cloud calls. Proven live: with the newline stripped the operator registers all 14 preset scripts (incl. ch-dc_snoop / ch-stack_trace / ch-creds_change) on the cluster. Signed-off-by: entlein <einentlein@gmail.com> * fix(ae): ASCII-only comments in stack_trace.pxl preset The arc `mypy` linter (mypy 1.20.2, `show_column_numbers = True`) crashes with an INTERNAL ERROR on stack_trace.pxl in CI. The file was the only preset carrying a multibyte character the linter had not seen before — a `->` (U+2192 arrow) in a comment; no other .pxl in the tree uses one. With column reporting on, mypy's byte-vs-character column bookkeeping over a multibyte comment char is a known crash vector. The sibling presets that lint clean are ASCII (dc_snoop) or carry only an em-dash that does not trigger it (creds_change), so this narrows stack_trace.pxl's comments to ASCII to match the known-good profile. No script behaviour changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * fix(lint): disable mypy on-disk cache to stop concurrent-run crashes arc lint runs `mypy --config-file=mypy.ini <file>` once per file, in parallel. The three adaptive_export preset .pxl files (added together) are linted concurrently and share the incremental cache in the repo root; concurrent writers corrupt it, producing a nondeterministic mypy `INTERNAL ERROR` on whichever file loses the race (stack_trace.pxl on one run, dc_snoop.pxl on the next -- neither a content issue). Setting `cache_dir = /dev/null` makes each per-file invocation self-contained, so there is no shared cache to race on. Type-checking semantics are unchanged; per-file runs get no incremental benefit anyway. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * fix(lint): add license headers to AE presets + fix dc_snoop E127 Once the mypy cache crash was out of the way, arc lint surfaced two real violations on the three preset .pxl files: - missing Apache license header (all three) -- added the canonical header used by the other pxl scripts; - flake8 E127 (continuation line over-indented) on dc_snoop's pxtrace.UpsertTracepoint call -- collapsed to a single line, matching creds_change's call style. flake8 (.pxl.flake8rc) and mypy both clean locally on all three. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: native ClickHouse DSN for retention plugin (fixes vizier crash) The retention plugin's export sink is the query engine's native ClickHouseExportSink (clickhouse-cpp over TCP :9000), not the AE's own HTTP write path (:8123). It requires the DSN in native format clickhouse://user:pass@host:9000/db. Passing the AE's HTTP DSN (http://host:8123/db) made the sink parse "http" as the username and crash on connect, taking the whole vizier Unhealthy. - config: add NativeDSN() builder (native TCP port, no http scheme), distinct from DSN() which remains the AE's own HTTP write endpoint. - main: pass NativeDSN() to EnsureClickHousePluginEnabled, and retry the ensure up to 5x (the vizier plugin service can 404 for the first few seconds after boot, permanently skipping enablement otherwise). Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: dark-table schema must match the native export sink The retention plugin's ClickHouseExportSink (clickhouse_export_sink_node.cc) serializes Pixie columns with a fixed type mapping and auto-appends an event_time column as DateTime64(3) (milliseconds). The dark-vector tables were declared with types that don't match what the sink sends, so every INSERT threw TYPE_MISMATCH server-side and the clickhouse-cpp client segfaulted on ReceiveException — crashlooping kelvin. Corrected to the sink's actual output types: - time_ UInt64 -> DateTime64(9) (TIME64NS) - upid UInt128 -> String (UINT128 serialized as String) - pid Int32 -> Int64 (all Pixie ints are INT64) - old/new_uid UInt32 -> Int64 - event_time DateTime64(9) -> DateTime64(3) (sink auto-appends millis) Validated live: stack_trace exports 5609+ rows of real profiler data, 0 insert errors, kelvin stable, vizier Healthy. NOTE (follow-up): the protocol builtinPresetScripts tables (http_events/dns_events/conn_stats/pgsql_events + redis/mysql/cql/mongodb/ amqp/mux/tls_events) export through the same sink and have the identical event_time DateTime64(3) requirement, currently unmet — latent until those tables receive rows. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: presets emit event_time (nanos) so sink keeps DateTime64(9) The ClickHouse export sink auto-appends an event_time column as DateTime64(3) (milliseconds) ONLY when the exported df has a time_ column but no event_time column (clickhouse_export_sink_node.cc:186 `has_time_ && !has_event_time`). That millisecond column mismatches the tables' DateTime64(9) event_time and crashes the native client on INSERT. Rather than degrade every table to DateTime64(3) millis (which would break the nanosecond-consistent event_time contract shared with the AE HTTP write path and dx/soc joins — see the schema.sql header), each preset now sets `df.event_time = df.time_`. That makes the sink treat event_time as a normal TIME64NS column and emit DateTime64(9) nanoseconds, matching the schema. - dc_snoop.pxl / creds_change.pxl / stack_trace.pxl: add df.event_time = df.time_ - builtinPresetScripts (all protocol presets): same, before px.display - schema.sql: dark-table event_time reverted DateTime64(3) -> DateTime64(9) (the other sink-type fixes stay: time_ DateTime64(9), upid String, pid/uid Int64) Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: AE deploys dark-vector bpftraces at boot (owns tables + traces) Retention/cron export scripts cannot deploy a tracepoint — the cron executor drops the pxtrace mutation, so dc_snoop/creds_change tables were never created (Table not found forever), while stack_trace worked only because it's the native profiler (no tracepoint). The AE now owns tracepoint deployment. - script.DesiredTracepoints(): source of truth for the bpftraces the AE deploys (dc_snoop, creds_change; extend for V6/V8). Each has a <name>_deploy.pxl (import pxtrace + UpsertTracepoint, permanent TTL, idempotent upsert). - main.deployDesiredTracepoints: at boot (INSTALL_PRESET_SCRIPTS=true) run each deploy script as a mutation ExecuteScript over the pixie adapter, with retry. pxapi auto-sets Mutation:true for `import pxtrace`. - Split the export presets: dc_snoop.pxl / creds_change.pxl are now query+export ONLY (no UpsertTracepoint) — they read the already-deployed table. - CONTRACTS.md: C12 expanded to "AE owns schemata + table deployments + trace deployments"; new C16 (native-DSN export + nanos event_time) and C17 (AE deploys bpftraces, cron never does); boot diagram updated. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: confirm tracepoint deploy by table, not the mutation stream pxapi's result collector cannot decode the mutation-info response the vizier returns for a pxtrace deploy ("stream: unimplemented type"), so the deploy Query always errored even though the UpsertTracepoint applied server-side (verified: dc_snoop + creds_change reach RUNNING_STATE, dc_snoop exports 220k rows). The old loop treated that as failure → 5 redundant re-deploys + a false "could not deploy" warning per tracepoint, and delayed the next tracepoint. Now the deploy fires once, then confirms success by polling the tracepoint's OUTPUT TABLE (a plain DataFrame query: "Table not found" = not deployed; compiles = RUNNING, 0 rows ok). Re-fires the deploy every few attempts as a fallback. Real success/failure signal, no false negatives, no wasted re-deploys. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export/e2e: creds_change trace calibration (fire + verify in CH) End-to-end calibration for the creds_change dark-vector tracepoint (V7). Every run proves the two properties the trace exists for: a) the trace WORKS — the AE-deployed commit_creds bpftrace captures a real privilege escalation (REAL uid >0 -> 0), and b) attribution reaches ClickHouse — the event flows Pixie -> AE retention export -> forensic_db.creds_change carrying pid + comm. Fires deterministically with a stock python:3-slim Job, no custom image: setresuid(12345,0,0) drops the real uid to a sentinel while KEEPING euid=0 (privileged), then setuid(0) pulls the real uid back to 0 — exactly the commit_creds(new_uid==0 && old_uid>0) the tracepoint filters for. The sentinel old_uid=12345 makes the row unambiguous. Asserts the row lands with pid+comm; pod (pid->pod enrichment) is assert-or-log so it greens automatically later. Live+e2e gated (AELOAD_LIVE=1 AELOAD_E2E=1). VALIDATED live on rig 6a5fbc75 (aeprod38): count=1 pid=905034 comm=python3 old_uid=12345 new_uid=0, 25s e2e. Signed-off-by: entlein <einentlein@gmail.com> * fix(lint): gofmt config.go (align one-line clickhouse accessors) The NativeDSN() addition broke gofmt's alignment of the consecutive one-line accessor funcs; golangci's format check (gci/gofmt) flagged config.go. Pure gofmt -w, no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: pid->pod/namespace enrichment for dc_snoop + creds_change Dark-vector tracepoints emit a raw kernel pid with no upid, so the pod/namespace columns landed empty (unattributed cluster-wide firehose). The dc_snoop and creds_change export presets now resolve namespace+pod via the validated process_stats merge on pid (px.upid_to_pid, PodEnrichPxL join; pid-only, not pid+asid — on a dynamic tracepoint px.asid() is the aggregator asid). Best-effort left join: blank pod for host/transient pids (correct). The presets then select exactly the forensic_db columns so the merge's pid_x etc. don't reach the sink. This makes the dark tables filterable/steerable by pod, not just pid/comm. creds_change calibration updated: the escalation process now sleeps ~20s so process_stats samples its pid (a sub-second process is never attributed); the test verifies pod/namespace resolve to the calibration namespace when attribution lands (assert-or-log until proven stable live). CONTRACTS: +C18 (dark-vector pod attribution). Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: full k8s metadata enrichment (namespace/pod/container/node) Extends the dc_snoop/creds_change pid->pod enrichment to the full workload identity: the process_stats merge now also resolves container (ctx['container']) and node (px.upid_to_node_name), alongside namespace + pod. DDL adds a container column to both dark tables; presets select the exact column set so the merge's pid_x doesn't reach the sink. This gives forensic attribution the full k8s metadata (which container, in which pod, in which namespace, on which node) for every dark-vector event, not just pid/comm. creds_change calibration logs the full metadata and still asserts namespace resolves to the firing workload. CONTRACTS C18 updated. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: AE auto-creates dc_snoop/creds_change/stack_trace (canonical dark tables) The AE registered dx_dcsnoop/dx_creds (old buggy schemas: time_ UInt64, pid Int32, event_time DateTime64(3) millis, single uid) in builtinTables/OperatorOwnedTables/ KnownTables/darkVectorTables, but the export presets + tracepoints use dc_snoop/creds_change/stack_trace with the debugged, calibration-proven schemas (DateTime64(9) nanos, Int64, old_uid/new_uid, full k8s metadata). So the AE created tables the presets never wrote to and never created the ones they did — export hit "table not found" unless the tables were made by hand. Reconcile to the canonical names across every list (option A, matches the dx#129 alignment already requested of dx-agent): - builtinTables / OperatorOwnedTables / ddl.go KnownTables(x2): dx_dcsnoop -> dc_snoop, dx_creds -> creds_change, + stack_trace (V9 native profiler). dx_vfs_events/unlink/ dlookup/mprotect/bpf/ptrace kept, reserved for the bpftraces still to be written. - compile.go darkVectorTables: dc_snoop/creds_change (pid-merge enrichment); stack_trace resolves via upid, not listed. - schema.sql: dc_snoop/creds_change/stack_trace reformatted one-column-per-line (the schema-verify parser is line-oriented — multi-column lines silently dropped columns, which would fail VerifyPixieSchema at boot); + hostname on stack_trace; removed the superseded dx_dcsnoop/dx_creds blocks. - tests: builtinTables count 21 -> 22; dark-vector test names -> dc_snoop/creds_change. Validated live (rig 6a61314b, aeprod41 + manual tables): dc_snoop 140k rows with full namespace/pod/container/node metadata; creds_change calibration PASS with attribution namespace=creds-calib pod=creds-calib/... container=escalate node=cplane-01. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: enforce nanosecond-timestamp + single-auth invariants via tests Audit + guardrails so the two cross-cutting invariants can never silently regress. Timestamps (one unit = nanoseconds): - TestPixieTablesUseNanosecondTimestamps / TestNoMillisecondTimestampReintroduced assert EVERY pixie observation table (PixieTables) stores time_ + event_time as DateTime64(9), never DateTime64(3) millis. kubescape_logs (unix-ns UInt64 input) and alerts (kubescape millis) are non-pixie tables, excluded by construction. - The audit surfaced 6 not-yet-active dark tables (dx_vfs_events, dx_unlink, dx_dlookup, dx_mprotect, dx_bpf, dx_ptrace) still on the OLD millis schema (time_ UInt64, pid Int32, event_time DateTime64(3)); fixed to the canonical nanosecond shape (DateTime64(9), Int64) matching dc_snoop/creds_change so they cannot crash the native export sink when their bpftraces are written. Authentication (one method per context, no reinvention): - TestCloudClientAuthIsPixieAPIKeyHeader pins the cloud plugin client to the canonical "pixie-api-key" gRPC header (never bearer/JWT); RejectsEmptyKey forbids a silent unauthenticated fallback. - TestNoAuthReinvention walks the whole AE tree: every JWT goes through the shared jwtutils lib (GenerateJWTForService / SignJWTClaims / ParseToken) — no golang-jwt/dgrijalva/jwt.New/SignedString/jwt.Parse — and "pixie-api-key" lives only in internal/pixie. Exactly two surfaces: cloud=api-key, in-cluster=service JWT. Full AE suite: 15 packages green. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: dark tables carry consistent full k8s metadata (+ tests) The 6 not-yet-active dx_ tables were inconsistent with the canonical dc_snoop/creds_change beyond the timestamp scale: all six lacked the container column, and dx_vfs_events/dx_unlink lacked comm (yet dx#129's projector reads comm from them). Reconciled every dark table to the same attribution shape: comm + namespace + pod + container + hostname (stack_trace uses upid for identity). New guardrails (internal/clickhouse): - TestDarkVectorTablesHaveFullMetadata: every dark table has namespace/pod/ container/hostname — uniform workload attribution, nothing dropped at dx join. - TestDarkVectorTablesCarryProcessIdentity: every dark table has comm or upid. - TestDarkVectorSetMatchesPixieTables: the dark set stays inside PixieTables() so the nanosecond + metadata guards actually cover it. Full AE suite: 15 packages green. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: ASCII-only .pxl comments (lint — matches build-agent's stack_trace fix) Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: elect one pod for cluster-scoped setup (fix duplicate exports) RCA of ~35% duplicate rows in the dark tables (dc_snoop 69.6M rows / 45.1M distinct; the creds_change calibration event stored twice): the AE is a DaemonSet (one pod per node), but installPresetScripts + deployDesiredTracepoints are CLUSTER-scoped. Every pod registered them, so each preset got one duplicate cron script per node (observed: 28 cron scripts = 2x the 14 presets on a 2-node rig), and every dark table was exported once per node. Fix: elect a single deterministic leader — the AE pod on the lexicographically smallest node name (leaderNode). Every pod computes the same winner from the same DaemonSet pod list, so no lease/coordination is needed; the pod-list RBAC is already held (findVizierNamespace). Gate installPresetScripts + deployDesiredTracepoints on it. The node-local trigger/data-plane still runs on every pod. Fail-open on any k8s error (a transient duplicate beats skipping setup). Tracepoint deploy was already idempotent (UpsertTracepoint), so only the cron registration duplicated. leader_test.go pins the election: smallest node wins, deterministic, exactly one leader across all pods' identical views. Signed-off-by: entlein <einentlein@gmail.com> * fix(build): add k8s client deps to cmd/BUILD.bazel The leader-election commit added k8s.io/apimachinery, client-go/kubernetes, and client-go/rest imports to cmd/main.go but did not update the go_library deps, so GoCompilePkg failed on the AE image build. Adds the three deps (same labels the internal/config package already uses) -- what gazelle would generate. No source change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: bazel BUILD deps for the leader guard + invariant test targets aeprod44 release build failed: bazel "missing strict dependencies" for the k8s client-go imports the leader election added. Add them to cmd_lib (mirroring the config package): @io_k8s_apimachinery//pkg/apis/meta/v1:meta, @io_k8s_client_go// kubernetes, @io_k8s_client_go//rest. Also register the new invariant tests in their pl_go_test targets so bazel test runs (+ enforces) them: cmd_test (leader_test.go), pixie_test (auth_invariants_test.go), and metadata_/timestamp_invariants_test.go in clickhouse_test. The source-walk guard (TestNoAuthReinvention) now t.Skip's in a sandboxed build where the tree isn't on disk, instead of failing — the behavioral auth tests run everywhere. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: exclude our own monitoring/infra pods from dc_snoop + stack_trace The dark tables were dominated by SELF-OBSERVATION noise — Pixie's own pem/kelvin/ vizier-* + the AE (all in `pl`) + the forensic ClickHouse generate the bulk of dc_snoop dentry lookups and stack_trace profiler samples (69M+ dc_snoop rows on an idle rig, mostly pem/kelvin). That is not workload evidence. The dc_snoop + stack_trace export presets now drop rows whose resolved namespace is our stack's: pl, px-operator, olm, clickhouse, kube-system. The filter runs at the export, so the noise never lands in ClickHouse. Workload pods and host/kernel (blank-namespace) rows are retained. creds_change is left inclusive on purpose — a privilege escalation from our own components is a compromise signal, not noise, and it is low-volume. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: protocol presets export via px.export, not px.display The builtin protocol retention presets (dns_events, conn_stats, http_events, redis_events, …) used px.display(df, '<table>'), which relies on the retention plugin routing display output to forensic_db.<table>. Verified on a clean rig that this never writes: dns_events/conn_stats/http_events = 0 parts ever, while the DarkVectorPresets (dc_snoop/stack_trace) — which use px.export via the OTel ClickHouse sink — populate (dc_snoop 220k). Switch the protocol presets to the same px.export(px.otel.ClickHouseRows(table=…)) path so they write directly through the sink, self-contained, no plugin-routing dependency. Signed-off-by: entlein <einentlein@gmail.com> * dc_snoop.pxl: exclude host runtime + node-agent(honey) from export dc_snoop was ~99% self-observation + host on a live rig (k3s-server 295k, k3s-agent 218k, containerd-shim 193k, node-agent 55k rows). The export dropped only 5 namespaces and explicitly RETAINED host/blank-namespace rows -- which is the bulk of the noise. Now drop honey(node-agent)+kube-* namespaces AND the host/runtime/monitoring comms (k3s-*, containerd*, runc, kelvin, vizier-pem, systemd, iptables, kubelet). Kept in sync with dx benchlive.writeSelfExclusion. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: decouple tracepoint deploy from retention firehose DEPLOY_TRACEPOINTS (default on) deploys the bpftraces permanently, independent of INSTALL_PRESET_SCRIPTS. When INSTALL_PRESET_SCRIPTS is off, purge the operator-managed cron scripts so the cluster-wide export firehose stops and dx drives per-anomaly deduped export instead. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: trail fan-out watermark by QueryLag to stop losing sparse evidence Root cause: the per-table fan-out advanced its watermark to now each pass, but socket_tracer flushes rows a few seconds late. For a long-lived anomaly the watermark stays at ~now, so every sparse event (dns_events, dc_snoop) loses the flush race and is skipped forever, while continuous tables (conn_stats) always have fresh post-watermark rows and export fully. Fix: query up to now-QueryLag (default 30s, ADAPTIVE_QUERY_LAG_SEC) so late-flushed rows stay queryable. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: /export/start drives full steer-all export (dx steers, AE grabs all) In pull mode the control /export/start only Upsert'd the streaming activeSet — a no-op for the fan-out. Add controller.OrderExportAll (one-shot OrderQuery for every configured pixie table, concurrent, deterministic query_id for dedup) and have handleStart trigger it when a querier is wired. dx already calls StartExport default-on per referral with no triage gate, so this makes dx steer AE to capture the COMPLETE evidence set for each anomaly's pod — filtered only to namespace/pod. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: floor OrderExportAll per target (stop dx StartExport flood) dx fires StartExport on every referral (~1s floor), so a sustained attack made OrderExportAll re-run the full 20-table capture many times/sec for the same pod over overlapping windows — the broker's globalSem saturated and nothing completed (dx-steered exports wrote 0). Add a per-target ExportAllFloor (ADAPTIVE_EXPORT_ALL_FLOOR_SEC, default 30s): one full capture per target per floor; the rolling window still covers every event. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: wire EXPORT_MODE=never to actually disable trigger self-steer EXPORT_MODE was validated but never consumed — the kubescape trigger self-steered regardless, so EXPORT_MODE=never was a no-op and the AE could not be put in a dx-only export mode. Add Config.DisableSelfSteer (set by EXPORT_MODE=never in main.go, inverted bool so the zero value preserves legacy self-steering), and gate the trigger's pushPixieRows spawn (handle + Rehydrate) on it. The control surface (dx /export/start OrderExportAll, /query OrderQuery) is unaffected — so with EXPORT_MODE=never the AE exports ONLY what dx steers. Signed-off-by: entlein <einentlein@gmail.com> * chore: remove stray 91MB cmd binary accidentally committed at repo root A compiled ARM aarch64 ELF executable named `cmd` (91MB) was committed at the repo root -- accidental `go build`/bazel output, not gitignored. It bloats the tree and would trip filename/artifact lint. Removed; no source or BUILD change (the real binary target is //src/vizier/services/adaptive_export/cmd). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * fix(lint): gofumpt presets.go (blank line between var blocks) golangci-lint's gofumpt formatter flagged presets.go:16 -- gofumpt wants a blank line between the two consecutive top-level var blocks (defaultExcludeNamespaces / defaultExcludeComms). Applied gofumpt -w to that file only. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * fix(lint): gci/gofumpt main.go + rename min var to avoid shadowing builtin golangci flagged main.go: (1) gci/gofumpt formatting — the new env-var consts and controller.Config fields broke alignment; (2) predeclared — leaderNode's local var `min` shadows the Go 1.21 builtin. Reformatted with gci+gofumpt (repo sections standard/default/prefix(px.dev)) and renamed the var to `smallest`. Builds clean; no behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * fix(lint): add pl_go_test target + E265-clean dc_snoop sentinel Two CI lint failures from the config-driven-presets work: - internal/script/BUILD.bazel: presets_test.go was added without the gazelle-generated pl_go_test target -> 'Gazelle was not run'. Added the script_test target (matches gazelle diff + cmd/BUILD.bazel style). - dc_snoop.pxl: the runtime sentinel '#__DC_SNOOP_EXCLUSION__' tripped flake8 E265 (block comment needs '# '). Renamed to '# __DC_SNOOP_EXCLUSION__' and updated the matching strings.Replace target in presets.go so the env-injected exclusion still substitutes byte-for-byte. No behavior change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: node-scope dark-vector tables so the malignant evidence lands The dark tables (dc_snoop/creds_change/dx_*) came back EMPTY: the AE filtered them by pod, but an incident's transient malignant pids (whoami/cat/getent children) are too short-lived to enter process_stats, so their ns/pod resolves blank and the filter dropped exactly the evidence. The AE is node-local (pem-direct → the node's own PEM), so QueryFor now keeps every dark row in the window and only drops the infra/self comms (DC_SNOOP_EXCLUDE_COMMS, env-tunable) — the workload's dark activity is captured node-scoped, no relevance filtering. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: default to pem-direct (node-local PEM :50305) pem-direct is the robust query path for the node-scoped AE: node-local (matches its scope), desync-immune (bypasses the kelvin/broker aggregation that the recurring PEM desync silently breaks) and fast. Default to HOST_IP:50305 when the deploy provides HOST_IP (downward API) + PL_JWT_SIGNING_KEY; explicit ADAPTIVE_VIZIER_DIRECT_ADDR still wins; cloud passthrough only as fallback. Signed-off-by: entlein <einentlein@gmail.com> * chore: remove re-added 91MB cmd binary + gitignore /cmd The stray adaptive_export ELF binary got committed at the repo root again (a repo-root `go build` output). Removed it and added `/cmd` to .gitignore so it stops recurring. No source/BUILD change; the real binary target is //src/vizier/services/adaptive_export/cmd. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * chore: gitignore /cmd (durable fix for recurring root binary) Follow-up to fd9108a, which removed the binary but did not land the .gitignore rule. Adds `/cmd` so a repo-root `go build` output stops getting re-committed. No source/BUILD change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: tracepoints via broker-direct on pem-direct + async_insert=0 writes Two fresh-rig blockers for the full evidence set: 1. pem-direct (:50305) serves fast node-local queries but REFUSES mutations (direct-query: mutations out of scope #29), so the bpftrace deploy failed and the dark tables (dc_snoop/creds_change/stack_trace) stayed empty. Deploy the tracepoints via the in-cluster broker (:50300) with the same JWT when the query adapter is pem-direct; keep querying via pem-direct. 2. Fresh ClickHouse defaults async_insert=1 → the AE's INSERT buffered and returned written_rows=0 (evidence trickled in minutes later / looked lost). Write with SETTINGS async_insert=0 so evidence lands + is counted immediately. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: stabilize dark-vector query (comm-filter before process_stats merge) The dc_snoop node-scope query timed out / silently dropped: it merged the node's ENTIRE dark stream (Formatter/vector/runc/... thousands of rows/window) against process_stats before filtering. Reorder: drop the infra/self comms FIRST, then merge — the pid-join now runs on the handful of workload rows so the dark capture completes reliably. Also expand the default comm-exclusion (runc:[2:INIT], CgrpMemUsgObsr, Formatter, iptables-save, vector-worker, metrics-server, ...) in CODE, since the env value with '[' ':' breaks the PxL filter. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export/control: read steer timestamps as nanoseconds (pipeline unit) The control API (dx -> AE: /export/start t_end, /query window) carries unix timestamps, and the evidence pipeline's ONE unit is nanoseconds (event_time, dx referral windows). The handlers read them with time.Unix(v, 0) — i.e. as SECONDS — so dx's nanosecond t_end (~1.78e18) became a year-56-billion instant. Effect: every dx-steered OrderExportAll captured [t_end-600s, t_end] over that garbage instant, a window that overlaps NO data, so the full-evidence set (all dark tables: dc_snoop/conn_stats/creds_change/stack_trace, and every protocol table) silently returned zero rows on the dx-steered path. Only the self-steer path (its own now-based window) ever wrote anything, which is why redis_events populated but the dx-ordered dark capture never did. The same garbage window also flowed into ae_reconcile.win_start/win_end, overflowing the DateTime formatter and making CH reject the reconcile insert (HTTP 400) — which hid the whole failure from the reconcile instrument. Fix: read the control timestamps as nanoseconds, time.Unix(0, ns), matching the pipeline. No unit autodetection — one unit, everywhere. Test + struct doc-comments updated to nanoseconds. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export/pxl: bound dark-table process_stats scan to 2m (was 5m) The dark-vector pod-resolution merge scanned process_stats over a 5-minute window. On a busy node that table samples every live pid every ~10-30s, so the scan is large and is the dominant cost of the dark query — heavy enough that, sharing the fan-out's query-slot budget with the fast native-table queries, the dark capture either starved (too few slots) or, once the steer windows were real, saturated the node-local PEM (too many). A 2-minute window still resolves the pods that matter — long-lived workload pids (redis-server) are sampled continuously — while transient attack pids never enter process_stats and resolve blank either way. Cuts the merge scan ~2.5x so the dark tables complete within the fan-out budget alongside the native queries. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export: dedup evidence tables via ReplacingMergeTree on natural event keys The evidence tables were plain MergeTree, so the dx-steered OrderExportAll — which re-pulls a rolling 600s window every ExportAllFloor — re-inserted the SAME kernel/ protocol events as fresh rows on every overlapping capture. dc_snoop showed 4,323 rows for one redis incident where the true unique count is a fraction of that. Switch the seven evidence tables to ReplacingMergeTree keyed on each event's NATURAL identity so re-pulls of the same event collapse: dc_snoop (time_, pid, comm, t, file, pod) creds_change (time_, pid, comm, old_uid, new_uid, pod) stack_trace (time_, upid, stack_trace_id, pod) redis_events (hostname, event_time, time_, upid, trace_role, remote_port, local_port, latency, req_cmd) dns_events (... , req_body) http_events (... , req_method, req_path) conn_stats (hostname, event_time, time_, upid, remote_addr, remote_port, trace_role) Keys are deliberately conservative — nanosecond time_ + pid/upid + connection tuple + a payload discriminator — so two DISTINCT events never share a key (false-collapse would drop real evidence, worse than a dup). Dedup is applied on background merge; exact counts read with FINAL / count(DISTINCT key). Bookkeeping tables already used ReplacingMergeTree; this brings the evidence tables in line. Signed-off-by: entlein <einentlein@gmail.com> * adaptive_export/pxl: source stack_trace from canonical stack_traces.beta The per-anomaly fan-out (QueryFor / OrderExportAll) queried px.DataFrame(table= 'stack_trace') — the ClickHouse table name, which is NOT a Pixie table — so the stack_trace evidence table never populated on the steered path. The native continuous profiler is 'stack_traces.beta' (upid-keyed, always-on, no tracepoint); only the retention preset used it, and that path is disabled. Fix: pixieSourceFor() maps stack_trace -> 'stack_traces.beta' for the DataFrame source (dotted-name DataFrames compile fine in a direct query — verified live), and a dedicated QueryFor branch resolves namespace/pod/container/hostname via ctx + upid_to_node_name and stamps event_time=time_, exactly like the export preset, then scopes to the alert's pod with the namespaced '<ns>/<pod>' key (ctx['pod'] is namespaced — verified live: pl/vizier-metadata-0 -> 112 samples). stack_trace is upid-native, NOT a dark-vector pid-merge table, so it takes the native resolution path, not the process_stats merge. Verified on a live cluster: the profiler emits across all pods, and the exact generated query returns real samples for a filtered pod. Signed-off-by: entlein <einentlein@gmail.com> --------- Signed-off-by: entlein <einentlein@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Entlein <eineintlein@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.