UnderCurrent

A Risk-Aware Autonomic Control Plane — monitors containerised workloads at the kernel level with eBPF, scores failure risk in real time, and triggers corrective actions without human intervention. Two parallel tracks: Stateless (Type S) and Stateful (Type F).

Authors: Pranav Negi · Reema Sarkar Paper: eBPF-Driven Dual-Track Control Plane for FSM-Gated Autonomous Remediation of Containerized Workloads — source in paper/uc.tex

---

At a glance

Safety ablation	Over 6,735 real Type-F decisions, the FSM-gated controller issued 0 premature and 0 missed checkpoint-restarts. Removing the FSM fires 314 premature restarts; threshold-only control misses all 319.
Recovery (MTTR)	30/30 injected I/O faults recovered per workload: postgres 10.95 s, redis 10.43 s, mysql 11.66 s. A naive restart-on-signal controller is ~5 s faster because it skips the verified durable flush.
Trace corpus	27,592 traces from a controlled fault campaign on four containers (postgres, redis, mysql, nginx), wasmtime sandbox active throughout — data/phase3/
Policy sandbox cost	27 µs mean, 50 µs p99 per WASM policy check (N = 10,000)
Checkpoint/restore	CRIU checkpoint creation works for postgres and redis; restore fails in Docker's experimental checkpoint path, so recovery falls back to docker restart

Every number is sourced in results/measurements.md; the corpus metrics are regenerated by scripts/finalize_corpus.py.

---

How it works

Every reconcile cycle (default 5 s) runs two parallel paths:

┌──────────────────────────────────────────────────────────────────────────┐
│                           UnderCurrent                                   │
│                                                                          │
│   ── Stateless Track (Type S) ──────────────────────────────────────     │
│                                                                          │
│   Kernel events          StateStore        Confidence                    │
│   (eBPF / simulated) ──▶ 60s window    ──▶ Scorer (0.0–1.0)              │
│                                                    │                     │
│                                          ┌─────────▼──────────┐          │
│                                          │  Reconcile DAG     │          │
│                                          │  restart/reschedule│          │
│                                          └──────┬──────┬──────┘          │
│                                           [real]│  [shadow]              │
│                                                 ▼      ▼                 │
│                                          Actions   Shadow log            │
│                                                 │                        │
│                                          stateless/traces.jsonl          │
│                                                                          │
│   ── Stateful Track (Type F) ───────────────────────────────────────     │
│                                                                          │
│   Kernel events          StatefulStateStore   Confidence                 │
│   (I/O, VFS, mounts) ──▶ 60s window       ──▶ Scorer (I/O signals)       │
│                                                    │                     │
│                                          ┌─────────▼──────────┐          │
│                                          │ FSM-gated reconcile│          │
│                                          │ flush / checkpoint │          │
│                                          │ / escalate         │          │
│                                          └──────┬──────┬──────┘          │
│                                  WASM policy ◀──┤      │                 │
│                                           [real]│  [shadow]              │
│                                                 ▼      ▼                 │
│                                          Actions   Shadow log            │
│                                                 │                        │
│                                          stateful/traces.jsonl           │
└──────────────────────────────────────────────────────────────────────────┘

• Real path — decisions are computed and executed; written to traces.jsonl with "mode": "real".
• Shadow path — decisions are computed against a fresh, memoryless FSM and never executed; written with "mode": "shadow". Any difference from the real path is logged as [DIVERGENCE]. Collect a corpus with --mode divergence — --mode real runs no shadow path.

Kernel signals

Track	Probe	Event
S	tracepoint sched:sched_process_exit	process_exit
S	kprobe tcp_connect	tcp_connect
F	kprobes blk_account_io_start / blk_account_io_done	blk_io_latency — keyed by request pointer, attributed to the issuing task
F	kretprobes vfs_write / vfs_read	vfs_write_error / vfs_read_error — storage errnos only (EIO, ENOSPC, EROFS, …), not EAGAIN/EINTR
F	tracepoints sys_enter_mount, sys_exit_mount, sys_enter_umount	volume_mount_restored / volume_mount_lost

Events are mapped to containers by process name, including server thread names (ib_*, connection → mysql; bio_* → redis).

---

Decision logic

Stateless (Type S)

Confidence score	Action
< 0.80	no_action
0.80 – 0.94	restart (docker restart <container>)
≥ 0.95	reschedule (logged intent only — no orchestrator)

A process exit scores 0.95; TCP connects scale to 0.90 at five events in the window.

Stateful (Type F) — FSM-gated

Confidence score	Raw action	FSM gate
< 0.50	no_action	—
0.50 – 0.79	flush_io_queue	reversible, always allowed
0.80 – 0.94	checkpoint_and_restart	only from Audited; otherwise downgraded to a flush
≥ 0.95	escalate	only from Degraded / Audited / Repairing
any	volume_delete	irreversible — always blocked

FSM lifecycle: Healthy → Degraded → Audited → Repairing → Recovered → Healthy

A failing database therefore takes at least two cycles to reach a restart: the first flushes and audits, the second may checkpoint and restart.

What each action does

Action	Effect
flush_io_queue	A real, synchronous durable flush inside the container via docker exec: postgres CHECKPOINT, redis SAVE, mysql FLUSH TABLES; FLUSH LOGS. Verified by scripts/verify_flush.py.
checkpoint_and_restart	docker checkpoint create (CRIU) → docker start --checkpoint → falls back to docker restart. Failed checkpoint images and containerd staging directories are cleaned up.
escalate	Logged intent only — no pager integration.

Two independent gates. After reconcile() applies the FSM rules, execute() re-checks every real action in a WebAssembly sandbox (stateful/wasm/*.wat, run by wasmtime). The sandbox is checked against fsm_state_at_dispatch — the state that authorised the action, before approve_repair moves the FSM to Repairing. If wasmtime is not installed, a Python function with the same rules is used and a warning is printed.

---

Results

Safety ablation — data/phase3/ablation_results.json

Scores held fixed, decision mechanism varied, N = 6,735 real Type-F records.

Controller	Checkpoint-restarts	Premature	Missed
Proposed (full FSM)	319	0	0
No FSM (always restart)	633	314	0
Threshold only (= shadow)	0	0	319
No confidence (binary)	0	0	319

Recovery — results/mttr_2026-09-01/

Fault injection to completed checkpoint_and_restart, 30 episodes each, 0 timeouts. Faults are 8 × 50–54 ms blk_io_latency events injected into the state store (not through eBPF).

Workload	UnderCurrent	Naive restart-on-signal
postgres	10.95 s (σ 0.06)	5.80 s (σ 0.05)
redis	10.43 s (σ 0.03)	5.35 s (σ 0.02)
mysql	11.66 s (σ 0.13)	6.54 s (σ 0.10)

The ~5 s difference is one reconcile interval: the flush-and-audit cycle that the naive controller skips.

Corpus metrics — data/phase3/metrics.json

Controlled fault campaign, 2026-09-02: 60 s injections rotating across postgres (COPY to /dev/full, ENOSPC), redis and mysql (writes under a host-side direct-I/O noisy neighbour) and nginx (worker kill), 12 slots per container, controller in divergence mode with wasmtime active.

Metric	Value
M1 cross-track contamination	0.0000 (0 of 27,592 traces)
M2 divergence, Type F	0.0474 (319 / 6,735)
M4 reversibility	1.0000 (2,944 active decisions)
M5 explanation completeness	1.0000
M6 FSM deferrals, Type F	0.0466 (314 / 6,735)

The earlier single-container snapshot is kept in data/phase2/.

Overhead — scripts/benchmark_overhead.py

WASM policy check	27.2 µs mean · 26.1 p50 · 29.6 p95 · 50.4 p99
Shadow path	+8.1 µs per reconcile (6.9 → 14.9 µs)
CRIU checkpoint creation, postgres	405.4 ms (σ 31.6), 30/30 created

---

Known limitations

• Checkpoint restore does not work under Docker. Postgres and redis checkpoints are created, but docker start --checkpoint fails on a network-namespace bind mount (containerd#12141); mysql fails during the dump. Every measured recovery is a docker restart after a verified flush.
• MTTR faults are synthetic. Episodes inject events into the state store, so MTTR excludes eBPF detection latency, and ends at the completed restart, not service readiness. eBPF probe overhead is not measured.
• Restarts repeat while a fault persists. Under a continuous fault, checkpoint-restart fires roughly every 10 s; there is no backoff.
• Audited has no exit when a fault clears. A container stays Audited, and a later fault goes straight to checkpoint-restart. This is why M2 and M6 differ slightly in data/phase3.
• The WASM sandbox blocked nothing in the corpus. The FSM gate defers every ineligible action first, so M6 counts FSM deferrals, not sandbox blocks.
• The corpus's Type-S track saw only process_exit, which scores 0.95 and triggers reschedule (simulated). The restart branch and tcp_connect signal are not exercised in data/phase3.
• escalate, reschedule are log-only, and containers are identified by process name rather than cgroup.
• One host, one instance per workload.

---

Repo structure

UnderCurrent/
├── stateless/                   ← Type S control plane
│   ├── main.py                  orchestrator — full pipeline
│   ├── event_listener.py        eBPF: process exit, tcp_connect
│   ├── state_store.py           60s sliding-window failure tracker
│   ├── confidence.py            risk scorer (0.0–1.0)
│   ├── reconcile.py             decision DAG + shadow controller
│   ├── actions.py               restart / reschedule executor
│   └── trace_logger.py          append-only JSON-line audit log
│
├── stateful/                    ← Type F control plane
│   ├── main.py                  orchestrator — full pipeline
│   ├── event_listener.py        eBPF: block I/O latency, VFS errors, mounts
│   ├── state_store.py           StatefulStateStore — extended event schema
│   ├── fsm.py                   ContainerFSM — formal state machine
│   ├── confidence.py            risk scorer (I/O latency, VFS, mount signals)
│   ├── reconcile.py             FSM-gated decision engine + shadow controller
│   ├── actions.py               durable flush / checkpoint-restart / escalate
│   ├── ablation.py              decision-policy ablation replay
│   ├── trace_logger.py          append-only JSON-line audit log (extended schema)
│   ├── wasm_executor.py         WASM policy sandbox (wasmtime)
│   └── wasm/                    WAT policy modules, one per action
│
├── shared/                      ← cross-track contracts and metrics
│   ├── constants.py             thresholds and window, shared by both tracks
│   ├── trace_schema.py          field catalogue + schema version (1.4.0)
│   ├── metrics.py               research metrics M1–M6
│   ├── eval_runner.py           compute metrics from trace files
│   ├── verify_safety_gates.py   design-claim verification
│   ├── cross_eval.py            cross-controller comparison harness
│   └── sensitivity.py           trace-replay threshold sweep
│
├── evaluation/                  ← scenario-based evaluation suite
│   ├── fault_harness.py         12 fault scenarios
│   ├── baseline_controller.py   naive threshold baseline
│   ├── run_trials.py            multi-trial experiment runner (N=30)
│   ├── compute_metrics.py       confusion matrices + pairwise tests
│   ├── distribution_analysis.py MTTR distribution analysis
│   ├── ablation.py              ablation study
│   ├── sensitivity.py           threshold sensitivity sweep
│   └── results/                 committed CSV/JSON results
│
├── scripts/                     ← real-container experiments
│   ├── run_episodes.py          MTTR episodes (--controller undercurrent|naive)
│   ├── inject_fault_v2.sh       real fault injection per workload
│   ├── fault_campaign.sh        rotate injections across all four containers
│   ├── gate_injector.py         check each injector actually trips the scorer
│   ├── launch_corpus_run.sh     one-shot corpus collection launcher
│   ├── finalize_corpus.py       snapshot a run and compute every paper metric
│   ├── verify_flush.py          prove flush_io_queue moves a durability counter
│   └── benchmark_overhead.py    WASM and shadow-path overhead
│
├── data/phase2/  data/phase3/   ← frozen trace corpora (phase3 is current)
├── results/                     ← measurements.md + MTTR summaries
├── paper/uc.tex                 ← conference paper source
├── integration/                 ← supervised launcher + Streamlit dashboard
├── dashboard/                   ← Flask API + React (Vite, Tailwind) frontend
├── docker-compose.yml           ← the four workloads, pinned by image digest
├── run.py / shadow.py           ← entry points — real / shadow mode
└── streamlit_dash.py            ← entry point — Streamlit dashboard

---

Quickstart

Run everything in simulate mode

git clone https://github.com/825pranav/UnderCurrent.git
cd UnderCurrent
pip install -r requirements.txt flask flask-cors
python3 run.py

Starts both pipelines with simulated events and the React dashboard at http://localhost:5050. For the Streamlit view instead: python3 streamlit_dash.py.

Reproduce the paper's metrics

# Corpus metrics M1, M2, M4–M6 from the committed snapshot
python3 shared/eval_runner.py --data-dir data/phase3

# Ablation replay on the same corpus
cd stateful && python3 -c "import ablation; ablation.run(trace_file='../data/phase3/stateful_traces.jsonl', out_file='/tmp/ablation_phase3.json')" && cd ..

# Scenario trials, sensitivity, figures
python3 evaluation/run_trials.py
python3 evaluation/sensitivity.py
python3 figures/generate_figures.py

stateful/traces.jsonl and stateless/traces.jsonl are gitignored — they grow as the system runs. Use --data-dir to compute metrics from a frozen snapshot.

Real-container experiments (Linux, root, Docker with experimental mode)

docker-compose.yml pins the exact images that were measured (postgres 18.3, redis 8.6.2, mysql 9.6.0, nginx 1.29.6) and declares the postgres and mysql data volumes as external, so on a fresh machine create volumes with those names first, or edit the volumes: block.

docker compose up -d
python3 scripts/verify_flush.py                        # flush gate
python3 scripts/run_episodes.py --container postgres --episodes 30
python3 scripts/run_episodes.py --container postgres --episodes 30 --controller naive

Corpus collection (scripts/launch_corpus_run.sh, then scripts/finalize_corpus.py) is documented in HANDOFF_v4.md.

Run tracks independently

cd stateful
python3 main.py                      # real mode, simulated events
python3 main.py --mode shadow        # shadow only — nothing executes
python3 main.py --mode divergence    # both paths, log disagreements
python3 main.py --interval 10        # reconcile every 10 s (default 5 s)

The stateless track takes the same flags, plus --rate for the simulated event rate.

Real eBPF mode (Linux only — requires root)

# Ubuntu / Debian
sudo apt-get install bpfcc-tools python3-bpfcc linux-headers-$(uname -r)
sudo python3 run.py --real --mode divergence

On Windows consoles, set PYTHONIOENCODING=utf-8; the controllers print arrow characters.

---

Dashboards

React dashboard

python3 run.py
# http://localhost:5050

Overview, Stateless, Stateful, Shadow and Audit Log tabs; real/shadow filter, score timeline, action distribution, FSM state cards, divergence table, sortable audit log, CSV and PDF export. Polls the Flask API every 4 s.

Streamlit dashboard

python3 streamlit_dash.py
# http://localhost:8501

Both tracks side by side — score timeline, action distribution, FSM state panel, research metrics and unified audit log. Auto-refreshes every 4 s.

---

Audit log format

Both tracks write their own traces.jsonl. Base fields are shared; stateful traces carry additional fields.

Base fields (both tracks):

{
  "trace_time": 1774186710.5,
  "container": "nginx",
  "score": 0.95,
  "action": "reschedule",
  "mode": "real",
  "why": "score 0.95 >= reschedule threshold 0.95",
  "executed": true,
  "stdout": "...", "stderr": "",
  "decision_timestamp": 1774186710.1,
  "action_timestamp": 1774186710.4
}

Additional fields in stateful traces:

{
  "node_type": "F",
  "fsm_state": "Audited",
  "fsm_state_at_dispatch": "Audited",
  "fsm_state_after": "Repairing",
  "fsm_transition": "Audited→Repairing",
  "reversibility": "reversible",
  "kernel_signals": ["blk_io_latency_high"],
  "dag_pattern": "io_degradation_major",
  "blocked_reason": null,
  "wasm_blocked": false,
  "wasm_reason": "WASM sandbox: policy satisfied"
}

Schema version 1.4.0 — see shared/trace_schema.py.

---

Requirements

• Controllers, simulate mode: Python 3.9+
• Dashboards: streamlit, pandas, plotly, flask, flask-cors
• WASM sandbox: wasmtime (falls back to an equivalent Python policy if absent)
• Real eBPF mode: Linux, root, bcc system package
• Actions and experiments: Docker daemon; checkpoint needs "experimental": true in /etc/docker/daemon.json

See requirements.txt for details.

---

License

MIT — see LICENSE