Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .claude/skills/ci-gates/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -1697,3 +1697,50 @@ this campaign has twice found a written-down limitation to be invented, and a
declared exception costs every future reader more than a case costs to write
once.

## 43. Testing a published prediction, and what the attempt found instead

The twelfth post claims that controls miss *degenerate* inputs — the empty case,
the equal case, the one off the edge — because a control author plants examples
**of** a fault, and degenerate inputs are examples of nothing. It says outright
that this predicts a hole in other suites and does not measure one.

So: measure one. `tri gates mutate` learned `--dir`, and the campaign's own
question was pointed outside its repository for the first time.

**The prediction could not be tested, and the reason is a finding.** The second
repository has **54 Python tools and not one declared negative control** — no
`--self-check`, no `--selftest`, under any spelling. There is nothing to ask
"did the control notice?" of. That is the state the first repository was in
before this campaign, and it is a larger fact than the prediction was.

The only other controlled gate in reach is the one I wrote yesterday, *from
these lessons*. Its boundary column reads `0/0` — the file contains no
comparison at all, so the prediction is untestable there too, honestly and for a
boring reason.

**The run found something else.** That gate — nine control cases, written with
more discipline than anything else this week — had a surviving silent mutant:
`main()`'s **no-argument** branch. Every case passes a build path, so nothing
ran it the way a broken *caller* would.

It matters because of where the gate is wired. The unattended publisher calls it
inside an `if !`, so a lost argument reaching `return 0` reads as a passing check
and the cron publishes having compared nothing. **The vacuous pass, inside the
guard written against the loss it exists to prevent.**

And beside it: the usage line was `__doc__.splitlines()[-4]`, which printed the
`--history` line rather than the usage line. A message whose content is an index
into the prose above it is wrong the moment the prose is edited — and it already
was.

**Two rulers widened to make the run possible, both worth keeping.** The file
filter matched `check_` and not `check-`, so aimed elsewhere it found nothing
and printed an empty table, which reads exactly like a clean suite. And `code()`
hard-coded `tools/`, so with `--dir` it would have run a *different file of the
same name* or none. Both are the shape this campaign keeps meeting: an
instrument that works because everything it has ever seen was arranged one way.

**The rule.** A gate you wrote from the lessons is still a gate. Point the
auditor at it, especially then — and when a prediction cannot be tested, say
which precondition failed rather than letting the attempt read as a confirmation.

94 changes: 75 additions & 19 deletions cli/tri/src/gates.rs
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,19 @@ pub enum GatesCmd {
#[arg(long)]
invert: bool,

/// Audit gates in this directory instead of `<repo>/tools`.
///
/// The gate SCRIPTS refuse a path override on purpose -- a flag that
/// aims a live gate somewhere harmless is a way to make it pass. This
/// is the auditor, not a gate: pointing it at another repository is
/// the whole job, and refusing to would mean every lesson this command
/// has learned stays inside one tree.
///
/// The dirty-tree refusal follows the directory, so an interrupted run
/// is still recoverable with `git checkout` in the right repository.
#[arg(long)]
dir: Option<String>,

/// Run all three operators in one pass and print them as columns.
///
/// Three commands answering one question is the same shape as two
Expand Down Expand Up @@ -131,7 +144,13 @@ fn sweep(controls_only: bool) -> Result<()> {
.filter_map(|e| e.ok().map(|e| e.path()))
.filter(|p| {
let n = p.file_name().and_then(|s| s.to_str()).unwrap_or("");
n.ends_with(".py") && (n.starts_with("check_") || n.contains("gate"))
// Both separators. The hyphen was not an oversight worth ignoring:
// aimed at another repository with --dir, the underscore-only
// filter silently found nothing and reported an empty table, which
// reads exactly like a clean suite. Two naming conventions for one
// thing is a shape this campaign has met before.
n.ends_with(".py")
&& (n.starts_with("check_") || n.starts_with("check-") || n.contains("gate"))
})
.collect();
files.sort();
Expand Down Expand Up @@ -167,7 +186,7 @@ fn sweep(controls_only: bool) -> Result<()> {
let gate = if controls_only {
"-".to_string()
} else {
code(&root, &name, &[])
code(&root, &root.join("tools"), &name, &[])
};
let ctrl = if flags.is_empty() && external.is_none() {
uncontrolled.push(name.clone());
Expand All @@ -179,11 +198,11 @@ fn sweep(controls_only: bool) -> Result<()> {
// not excuse a red sibling.
flags
.iter()
.map(|fl| code(&root, &name, &[fl]))
.map(|fl| code(&root, &root.join("tools"), &name, &[fl]))
.find(|c| c != "0")
.unwrap_or_else(|| "0".to_string())
} else {
code(&root, external.as_ref().unwrap(), &[])
code(&root, &root.join("tools"), external.as_ref().unwrap(), &[])
};
rows.push((name, gate, ctrl));
}
Expand Down Expand Up @@ -590,11 +609,35 @@ fn line_of(src: &str, byte: usize) -> usize {
src[..byte].matches('\n').count() + 1
}

fn mutate(only: Option<&str>, loud: bool, invert: bool, all: bool) -> Result<()> {
let root = repo_root()?;
fn mutate(only: Option<&str>, loud: bool, invert: bool, all: bool, dir: Option<&str>) -> Result<()> {
let (root, tools) = match dir {
None => {
let r = repo_root()?;
let t = r.join("tools");
(r, t)
}
Some(d) => {
let t = std::fs::canonicalize(d).with_context(|| format!("--dir {d}"))?;
// The gate runs with cwd at ITS repository root, not at the
// directory holding it: a gate that resolves paths relative to the
// working directory would otherwise be run somewhere it has never
// been run, and the difference would look like a finding.
let out = Command::new("git")
.args(["rev-parse", "--show-toplevel"])
.current_dir(&t)
.output();
let r = match out {
Ok(o) if o.status.success() => std::path::PathBuf::from(
String::from_utf8_lossy(&o.stdout).trim().to_string(),
),
_ => t.parent().map(|p| p.to_path_buf()).unwrap_or(t.clone()),
};
(r, t)
}
};
let dirty = Command::new("git")
.args(["status", "--porcelain", "--", "tools/"])
.current_dir(&root)
.args(["status", "--porcelain", "--", "."])
.current_dir(&tools)
.output()
.context("git status failed")?;
if !String::from_utf8_lossy(&dirty.stdout).trim().is_empty() {
Expand All @@ -605,11 +648,17 @@ fn mutate(only: Option<&str>, loud: bool, invert: bool, all: bool) -> Result<()>
);
}

let mut files: Vec<std::path::PathBuf> = std::fs::read_dir(root.join("tools"))?
let mut files: Vec<std::path::PathBuf> = std::fs::read_dir(&tools)?
.filter_map(|e| e.ok().map(|e| e.path()))
.filter(|p| {
let n = p.file_name().and_then(|s| s.to_str()).unwrap_or("");
n.ends_with(".py") && (n.starts_with("check_") || n.contains("gate"))
// Both separators. The hyphen was not an oversight worth ignoring:
// aimed at another repository with --dir, the underscore-only
// filter silently found nothing and reported an empty table, which
// reads exactly like a clean suite. Two naming conventions for one
// thing is a shape this campaign has met before.
n.ends_with(".py")
&& (n.starts_with("check_") || n.starts_with("check-") || n.contains("gate"))
})
.collect();
files.sort();
Expand Down Expand Up @@ -708,12 +757,12 @@ fn mutate(only: Option<&str>, loud: bool, invert: bool, all: bool) -> Result<()>
crate::mutate::clear_derived_caches(f);
let mut already_red: Vec<String> = Vec::new();
for fl in &flags {
if code(&root, &name, &[fl]) != "0" {
if code(&root, &tools, &name, &[fl]) != "0" {
already_red.push(fl.clone());
}
}
if let Some(c) = &external {
if code(&root, c, &[]) != "0" {
if code(&root, &tools, c, &[]) != "0" {
already_red.push(c.clone());
}
}
Expand Down Expand Up @@ -754,14 +803,14 @@ fn mutate(only: Option<&str>, loud: bool, invert: bool, all: bool) -> Result<()>
crate::mutate::clear_derived_caches(f);
let mut noticed = false;
for fl in &flags {
if code(&root, &name, &[fl]) != "0" {
if code(&root, &tools, &name, &[fl]) != "0" {
noticed = true;
break;
}
}
if !noticed {
if let Some(c) = &external {
noticed = code(&root, c, &[]) != "0";
noticed = code(&root, &tools, c, &[]) != "0";
}
}
// Restore before judging, so an early return can never leave the
Expand Down Expand Up @@ -883,9 +932,12 @@ fn mutate(only: Option<&str>, loud: bool, invert: bool, all: bool) -> Result<()>
Ok(())
}

fn code(root: &std::path::Path, script: &str, args: &[&String]) -> String {
fn code(root: &std::path::Path, tools: &std::path::Path, script: &str, args: &[&String]) -> String {
let mut c = Command::new("python3");
c.arg(format!("tools/{}", script));
// Absolute, so the script found is the one in `tools` rather than whatever
// sits under a directory of that name below cwd. With no --dir this is the
// same file the old relative form resolved to.
c.arg(tools.join(script));
for a in args {
c.arg(a.as_str());
}
Expand Down Expand Up @@ -923,9 +975,13 @@ fn repo_root() -> Result<std::path::PathBuf> {
pub fn run(cmd: &GatesCmd) -> Result<()> {
match cmd {
GatesCmd::Sweep { controls_only } => sweep(*controls_only),
GatesCmd::Mutate { only, loud, invert, all } => {
mutate(only.as_deref(), *loud, *invert, *all)
}
GatesCmd::Mutate {
only,
loud,
invert,
all,
dir,
} => mutate(only.as_deref(), *loud, *invert, *all, dir.as_deref()),
GatesCmd::Dead { repos, min_runs } => {
let list: Vec<String> = if repos.is_empty() {
["gHashTag/trinity", "gHashTag/trinity-fpga", "gHashTag/t27"]
Expand Down
16 changes: 16 additions & 0 deletions docs/now/2026-08-23-testing-a-published-prediction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# NOW — testing a published prediction (2026-08-23)

The twelfth post claims controls miss *degenerate* inputs and says outright that this predicts a hole in other suites and does not measure one. So: measure one. `tri gates mutate` learned `--dir`.

- **The prediction could not be tested, and the reason is the finding.** The second repository has **54 Python tools and not one declared negative control** — no `--self-check`, no `--selftest`, under any spelling. Nothing to ask "did the control notice?" of. That is the state the first repository was in before this campaign.
- The only other controlled gate in reach is the one I wrote yesterday *from these lessons*. Its boundary column reads `0/0` — no comparison in the file — so the prediction is untestable there too, honestly and for a boring reason.

**The run found something else.** That gate — nine control cases, more discipline than anything else this week — had a surviving silent mutant: `main()`'s **no-argument** branch. Every case passes a build path, so nothing ran it the way a broken *caller* would.

It matters because of the wiring: the unattended publisher calls it inside an `if !`, so a lost argument reaching `return 0` reads as a passing check and the cron publishes having compared nothing. **The vacuous pass, inside the guard written against the loss it exists to prevent.**

Beside it: the usage line was `__doc__.splitlines()[-4]` and printed the `--history` line. A message that is an index into the prose above it is wrong the moment the prose is edited — and it already was.

**Two rulers widened to make the run possible.** The file filter matched `check_` and not `check-`, so aimed elsewhere it found nothing and printed an empty table — which reads exactly like a clean suite. And `code()` hard-coded `tools/`.

**The rule:** a gate you wrote from the lessons is still a gate — point the auditor at it, especially then. And when a prediction cannot be tested, name the precondition that failed rather than letting the attempt read as a confirmation.
Loading