docker-sandbox-substrate: harden the isolation probe and model kit-supplied egress

[kyle-sexton]

Durable home for the decisions from the docker-sandbox-substrate interview. The
contract slice itself is working state (docs/topics/ is pruned before merge per
docs/conventions/topic-docs/README.md), so the outcome lands here until an
implementation PR can carry the PLAN in its body.

What was decided

A kernel-separated local isolation substrate was evaluated and probed live on
Windows 11 Pro under a hardened invocation (clone mode + a default-deny network
posture). Both ladder assertions failed inside the boundary — the passing
condition — with the outer context proving each target reachable or present
first.

Integrating it costs zero files in this repository: the sandbox-provider seam
in plugins/autonomy/reference/runner.md already exists, the guided-setup slice
already detects capability-shaped rather than instance-shaped, and
guardrails/isolation-ladder.md forbids naming a product instance here at all.
The instance's only legitimate home is the substrate field in a consumer's
security binding, outside this repo.

The work this leaves

1. Harden the probe's egress assertion. A connect-only test grades a
   microVM-class substrate wrongly: raw TCP connect() succeeds because the
   interception layer accepts the SYN and then drops the session, so TLS EOFs with
   no data. templates/isolation-probe.md currently leans toward "must fail to
   CONNECT" for metadata endpoints. The assertion must test whether data
   flows.
2. Add a third assertion covering the workspace mount. The existing two —
   denied egress, absent host credentials — do not test the bind-mounted
   workspace at all. A substrate can pass the probe cleanly while the agent can
   still write host-executing files (.git/config alone is a command key ring;
   core.fsmonitor executes host code on git status). The ladder's levels
   describe the process boundary; the workspace mount is a deliberate hole
   through it that nothing currently measures.
3. Decide whether the ladder models kit-supplied allow rules. During the
   probe a kit added its own network allow rule on top of a global deny-all.
   Installing a kit can widen egress. Unlike the other items this changes what a
   level binding certifies, and it is a supply-chain concern the ladder does not
   model.
4. Record a re-verification cadence for version-bound substrate evidence. The
   probed product is pre-1.0 and ships roughly every two weeks; specific flags,
   defaults, and guarantees rot. The durable findings are the architecture and the
   posture, not the specifics.

Why this is not a plugin change

Naming any product in this repository's contract surface would violate the
ladder's "classes, never vendors" rule. Items 1–3 are all class-level and
vendor-neutral.

No linked issue — this IS the tracking item.

Related

• melodic-software/dotfiles#427 (merged) — tracks the substrate CLI in the
  user-scope package list.

[kyle-sexton]

Items 3 and 4 resolved

Item 3 — kit-supplied allow rules: confirmed, and the ladder gains a class-level property

The premise was only ever half-verified. The rule's existence was confirmed; its effect never was — earlier probes sampled only hosts the kit did not allow, and the prior run recorded merely that openrouter.ai resolved, which by this work's own new standard is not evidence data flows.

Tested 2026-08-11 in a purpose-built sandbox, since torn down:

Host	Authorizer	HTTP	Peer issuer	Bytes read
openrouter.ai (kit-allowed)	Allowed	200	Google Trust Services — the origin's own CA	201,961
example.com (control)	Denied	403	Docker Sandboxes Proxy CA — the interceptor	118 (block page)

Governance mode was Local policy only throughout. A kit widened egress past a global default-deny and 200 KB of origin data flowed.

The vendor already solves this under organization governance, where the precedence table is kit allow ✗ / kit deny ✓ — "Precedence is decided by a rule's decision rather than its source." It does not solve it in local-policy-only mode, which the vendor's own documentation leaves unstated.

Decision: the ladder states a class-level property — an L2+ binding must assert that nothing the run can install can widen the boundary, only narrow it. Naming the vendor's component type would violate the ladder's classes-never-vendors rule and would not generalize; the property covers browser-extension permissions, admission controllers, and any additive policy engine.

Probe obligation this creates, which is easy to miss: target selection is load-bearing. A probe that samples only hosts the installed components do not allow will certify a boundary that is in fact open. The recipe must require probing in the configuration the run will actually use, including at least one target the installed components are permitted to reach.

Incidentally, the Phase 1 peer-identity assertion grades both rows correctly — origin CA for the widened host, interceptor CA for the denied one. It was validated against a genuinely open boundary, not only a sealed one.

Item 4 — re-verification cadence: event-triggered, with a staleness backstop

The framing did not survive the research. Release cadence is evidentially inert in both directions: CISA warns against reading fix counts as a negative signal; Ozment & Schechter measure median foundational vulnerability lifetime at ≥2.6 years with ~67.6% found after 7.5 years; Rescorla cannot exclude a constant discovery rate. Three weeks is not a sample, so a cadence keyed to release frequency would be ritual rather than control.

• Re-verify on events: a substrate version change touching the probed boundary; a policy-engine or governance-mode change; a change to the installed component set — which item 3 just proved can widen the boundary with no version change at all.
• Staleness bound: evidence older than this repository's existing ">2-month" idiom is stale regardless of events. Reusing that number rather than inventing one.
• Scope: re-verification covers the specifics that rot — flags, defaults, guarantees. Architecture and posture findings are durable and are not re-derived each time.

Status of the other items

Items 1 and 2 are implemented in #2150 (open, all checks green, awaiting human merge). Both decisions above land with the follow-on phases, because Phase 1 already edits isolation-ladder.md and a concurrent edit would conflict.

[kyle-sexton]

Items 1 and 2 are shipped

#2150 merged as c1b4c629; autonomy is at 0.15.0 on main.

• Item 1 — harden the egress assertion. Done, and the mechanism changed during the work: certificate verification turned out to be insufficient, not merely weaker. The measured boundary presents a certificate carrying the correct hostname signed by a CA it trusts, so verification returns success on a fully sealed boundary. Denial is now proven by comparing peer IDENTITY against the outer context, plus three supporting legs (client readiness, two distinct targets, recorded address families).
• Item 2 — third assertion covering the workspace mount. Done as workspace_host_write_contained, proven from the outer side so one rule covers both a read-only mount and a copy-on-read mount. Scope is deliberately WRITE containment, stated in the assertion's own name — read exposure is untouched and remains open.

Migration, already in the CHANGELOG: every level bound under the two-assertion recipe goes UNPROVEN until re-probed, which fail-closes autonomous dispatch on that surface. Intended, and breaking.

Items 3 and 4 are decided, not yet implemented

Both decisions are recorded in the comment above and in the PLAN on branch docs/docker-sandbox-substrate-plan. They land with the follow-on phases (verification topology), because implementing them means editing isolation-ladder.md and templates/isolation-probe.md again.

Item 3 carries a probe obligation worth restating, because it is easy to miss and #2150 does not yet satisfy it: target selection is load-bearing. A probe that samples only hosts the installed components do not allow will certify a boundary that is in fact open — measured at 201,961 bytes of origin data through a global default-deny.

Leaving this issue OPEN for items 3 and 4.

[kyle-sexton]

Why verification-topology.md carries no capability label — the sourced rationale

The contract leaf shipped in the verification-topology work states only that capability labels are rejected as policy vocabulary, with no such label written anywhere in it — not even as a rejected example, since a rejected-vocabulary section still puts the words in the normative artifact. This comment is where the per-label evidence lives, so a future reader tempted to reintroduce one has something to read.

Rejected	Why
frontier	Lands on non-top models at 3 of the 4 vendors that use it. Where it is rigorously defined (California SB 53) it denotes a 10^26 FLOP training threshold that never demotes — a different concept from "best current model", and disjoint from it in both directions.
flagship	OpenAI uses it for top-of-line; Google uses it for best-in-this-modality. Same word, incompatible referents.
daily driver / balanced	The nearest vendor-sanctioned form is Anthropic's sonnet alias, which resolves to different capability levels on different providers.
fast / cheap	The most consistent position across vendors, but every vendor's name for it differs and no cross-vendor resolution mechanism exists. Also dissolving intra-model: fast mode makes one model span two latency points.
reasoning vs instruct	A hard product binary at OpenAI; a dial inside every model at Anthropic and Google (effort, thinking). The axis is collapsing, not stabilizing.
A 4+ rung capability ladder	Zero empirical support. Every capability/difficulty router in the surveyed corpus is 2 rungs; the single 3-tier variant's middle tier is a cost region, not a role.

What the leaf ships instead: pipeline roles (properties of our pipeline, immune to roster churn by construction), relational constraints that bind a role by its relationship to another role, and predicates a binding can actually evaluate. not_weaker_than ships present but not defaulted — no cross-vendor capability ordering exists to evaluate it against, at any price.

Evidence behind the "independent aggregation, never deliberation" invariant

Also deliberately kept out of the contract: the leaf justifies that invariant on mechanism alone, because a single-study figure parked in a normative artifact strands the contract when a contrary result lands.

The measurement: a deliberative-consensus protocol scored ≈76% against 83.43% for independent aggregation with confidence-weighted voting over the same models — below every single-model baseline it was built from. Measured inter-model error correlation of 0.529–0.689 explains why aggregation gains stay small (~1.01pp over the best single model) even where aggregation wins.

Source: https://arxiv.org/abs/2605.30802. Confidence: T2, medium — single author, prediction-market domain. The corpus is not unanimous: 2305.14325 finds debate helps, and 2510.20963 finds deliberation helps if the protocol is structured. The invariant is written to survive that disagreement — it rests on the mechanism (agreement reached by discussion is correlation, not corroboration) rather than on the number, so a contrary result would demote the citation without silently contradicting a rule that had no stated warrant.

If a future change proposes relaxing the invariant or reintroducing a label, this comment is the thing to argue against.

[kyle-sexton]

Items 3–4 discharged — PR #2326

The verification-topology work is open as #2326, which closes this issue. What it settled, and what it deliberately did not:

Q21 — the ladder gained a class-level property, not a vendor-shaped rule. An L2+ surface's policy layer must let installed components narrow reachability and never widen it. The probe obligation this creates is the load-bearing half: target selection decides whether the egress assertion means anything, so the probe must cover every ratified component-reachable destination in FULL. Coverage, not at-least-one — each destination is a separate policy decision.

The ratified host set rides the level binding rather than the transcript or a CLI argument. A first design used a CLI seam; running it showed the no-hosts sentinel binding L2 with zero evidence, and the seam put the fact the check turns on where no reviewer of either the binding or the capture could see it.

Q22 — the three software-factory gaps are recorded as deferred-with-trigger, wording ratified as drafted: fleet-level economics, portfolio-scale multi-repo fan-out, self-service golden paths. Deferred, never rejected.

Q23 — event-triggered re-verification, not a release-frequency cadence. Release cadence is evidentially inert in both directions; re-verify on version, governance, or component-set change, with the repository's existing staleness idiom as backstop.

The finding worth carrying past this issue

Three times across two phases, in three different artifacts, the same defect: a count that does not guarantee the coverage it exists for. Phase 1's review round found four instances. Two independent audits of the contract leaf each found another — and the second was inside the repair for the first, where three deterministic slots satisfied a floor of three while the cross-vendor requirement bound an empty set and was vacuously true.

The generalization is now in the contract itself: whenever a rule counts things, ask what it would accept if every counted thing were identical.

Two further things surfaced only by executing the checks rather than reviewing them — a rule pair that was individually sound and jointly unsatisfiable, and a briefed check that could not be built at all because the surface it keyed on was floor-bounded and tighten-only. Neither was reachable by reading.