Skip to content

Execute the reference-process Sandbox harness (AB#847) #439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PolyphonyRequiem merged 2 commits into from
Sep 19, 2026
Merged

Execute the reference-process Sandbox harness (AB#847) #439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0377a1d
Execute the reference-process Sandbox harness (AB#847)
PolyphonyRequiem Aug 31, 2026
47750bd
docs(reference): bind sandbox evidence provenance AB#847
PolyphonyRequiem Sep 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 36 additions & 11 deletions docs/projects/reference-process-base-and-harness.plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,11 @@ Decisions locked here:
Twig has no plan verb for process customization). Every hierarchy/link seed step is
a Twig plan operation (`publish-seed`, `add-link`, `batch`).
4. **Harness structure:** a per-surface evidence bundle stored under
`docs/reference-process/harness/<version>/` in the Twig repo. Each of six required
surfaces has a named `.json` evidence artifact plus a `.png` screenshot artifact and
is produced by a specific plan operation against a dedicated `Twig-Reference-Sandbox`
ADO project.
`docs/reference-process/harness/<version>/` in the Twig repo. Each of the **ten**
required surfaces enumerated in §4.3 has a named `.json` evidence artifact, optionally
accompanied by a `.png` screenshot for the backlog and sprint surfaces, and is produced
by a specific plan operation against a dedicated `Twig-Reference-Sandbox` ADO project.
The `.json` artifact is authoritative — the gate reads JSON and never opens an image.
5. **Harness gating:** the harness is a **hard gate for committing a new version of the
reference profile** (owned by #732). It is **not** a runtime gate — Twig core still
validates the live process against the profile at runtime; the harness only guards
Expand Down Expand Up @@ -314,15 +315,15 @@ tailoring with the seed/link operations that exercise the Sandbox.
| 2 | Create the inherited-from-Basic process. | Human, REST | §2.1 |
| 3 | Disable `Epic` and `Issue`. | Human, REST | §2.2 |
| 4 | Add `Twig.Initiative`, `Twig.Investigation`, `Twig.Feature`, `Twig.Bug`. | Human, REST | §2.3 — must precede step 5 so behaviors have types to attach to. |
| 5 | Assign backlog behaviors. | Human, REST | §2.4 — must precede step 8 so sprint-only-for-Task is enforced before publish. |
| 5 | Assign backlog behaviors. | Human, REST | §2.4 — must precede step 8 so sprint-only-for-`Task` is enforced before the project is provisioned. |
| 6 | Verify required-field set. | Human, REST | §2.5 |
| 7 | Publish the process. | Human, REST or UI | §2.6 |
| 7 | ~~Publish the process.~~ **Not applicable.** | — | §2.6 — an inherited process is live on creation; no publish transition exists. Retained as a numbered row so steps 8–16 keep their identifiers. |
| 8 | Provision the `Twig-Reference-Sandbox` project. | Human, REST or UI | §2.7 |
| 9 | Confirm link kinds available. | Human, UI check | §2.8 |
| 10 | Record the baseline process description. | Automated (`twig process description`) | §2.9 |
| 11 | Populate the profile document from step 10's JSON (owned by #732). | Human, doc | #732 ratifies the profile against Sandbox reality. |
| 12 | Seed the hierarchy in Sandbox: one `Initiative`, one each of `Investigation`/`Feature`/`Bug`, three `Task`s. | **Twig plan** (`publish-seed`) | §4.1 |
| 13 | Link the hierarchy with parent/child, then predecessor/successor, then related, then artifact links. | **Twig plan** (`add-link`, `batch`) | §4.2 |
| 13 | Link the hierarchy with parent/child, then predecessor/successor, then related. | **Twig plan** (`add-link`) | §4.2 — one op per `workItemId` per plan file, so parent edges are expressed child-side and split across files. **Artifact links are not expressible as a plan op** and are applied out of band; see §4.2. |
| 14 | Capture per-surface evidence artifacts. | Automated (`twig show`, `twig tree`, headless browser) | §4.3 |
| 15 | Run the harness gate check. | Automated (script over the evidence bundle) | §5 |
| 16 | Ratify the profile document version. | Human, PR merge | §5 gating rule. |
Expand Down Expand Up @@ -439,8 +440,32 @@ ops:
linkType: ArtifactLink
```

The plan runs unchanged against a fresh Sandbox; every op resolves against alias-only
references so no work-item IDs are hardcoded.
🔴 **The YAML above is illustrative pseudocode, not a runnable plan file.** It was written
before the native surface existed. Corrected against the shipped contract while executing
this harness under AB#847 (see `docs/reference-process/harness/1.0.0/`):

- A real plan file is **proposal v1 JSON**: top level is exactly `version`, `workspace`
(`organization` + `project`), and `operations`. There is no `ops`, `seed`, `alias`,
`from`, `to`, or `linkType` key.
- Accepted kinds are `batch`, `add-link`, `remove-link`, `publish-seed`, `delete`. A
`publish-seed` op carries `stagedIdentity` + `expectedFingerprint` from
`twig proposal seed --id <negativeAlias> -o json`; a link op carries `workItemId`,
`expectedRevision`, `relation`, and `otherId`.
- `relation` is a **closed set**: `parent | predecessor | successor | related`.
**`ArtifactLink` is therefore not expressible as a plan operation** —
`twig proposal validate` rejects it with `plan.invalid_relation`. The surface was
applied directly against ADO REST, and the committed harness freezes the exact
project/repository/ref identity in `fixtures.json`; this is a tooling gap, not proposal
coverage.
- At most **one op per `workItemId` per plan file**. Parent edges must therefore be
expressed **child-side** (`workItemId` = child, `relation: parent`, `otherId` = parent)
and split across sequence files; the parent-side spelling above puts three ops on
`@INIT` in one file and is refused.
- Seeds are never referenced by real id or negative alias from a downstream op in the same
plan, so linking is necessarily a **later** plan file than seeding.

The seeding plan runs unchanged against a fresh Sandbox; the linking plans depend on the
ids the seed publish returned.

### 4.3 Required surfaces, evidence artifacts, capture recipe

Expand All @@ -454,8 +479,8 @@ references so no work-item IDs are hardcoded.
| 6 | Native parent/child rendering | `twig tree @INIT -o json` + `twig show @INIT -o json` | `06-hierarchy-links.json` | JSON shows `@INIT` → {`@INV`, `@FEAT`, `@BUG`} → {`@TA`, `@TB`, `@TC`}; every link's `rel` is `System.LinkTypes.Hierarchy-Forward` from the parent side |
| 7 | Native predecessor/successor rendering | `twig show @TA -o json` and `twig show @TB -o json` | `07-predecessor-successor.json` | `@TA` has a `System.LinkTypes.Dependency-Forward` link to `@TB`; `@TB` has the reverse |
| 8 | Native related rendering | `twig show @INV -o json` | `08-related-links.json` | `@INV` has a `System.LinkTypes.Related` link to `@FEAT` |
| 9 | Artifact link | `twig show @FEAT -o json` | `09-artifact-links.json` | `@FEAT`'s relations contain an `ArtifactLink` matching the seeded branch vstfs URI |
| 10 | Rank preservation across publish/link | `twig tree @FEAT -o json` immediately after step 12's task publish, again after step 13's link ops | `10-rank-before.json`, `10-rank-after.json`, `10-rank-diff.txt` | The child order under `@FEAT` in `before` equals `after` (task publish order preserved through subsequent link mutations). `diff.txt` is empty. |
| 9 | Artifact link | `twig show @FEAT -o json` | `09-artifact-links.json` | `@FEAT` is the only item and its relations contain a nonempty `ArtifactLink` whose URL matches the frozen project/repository/ref identity in `fixtures.json` |
| 10 | Rank preservation across publish/link | Server-owned backlog order for the portfolio and requirement backlogs (`GET /{project}/{team}/_apis/work/backlogs/{backlogId}/workItems`), captured after step 12's publish and again after step 13's link ops | `10-rank-before.json`, `10-rank-after.json`, `10-rank-diff.txt` | The two backlog orders are identical — rank survives publish and subsequent link mutation. `diff.txt` is empty. |

Every row in the table above corresponds to a surface #727 lists as "must be
observed". No surface is silently skipped. The evidence file names are canonical —
Expand Down
141 changes: 141 additions & 0 deletions docs/reference-process/harness/1.0.0/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# Reference-process validation harness — profile 1.0.0

Machine-checkable proof that the Twig reference process behaves as
`docs/projects/reference-process-base-and-harness.plan.md` (AB#733, "T2") specifies.
This bundle is a dated historical capture from a real ADO project on 2026-08-31 and is
frozen here as the artifact AB#727 means when it requires the reference process to be
"exercised against a Sandbox project before it is treated as authoritative".

**Status: PASS** — historical capture, all ten required surfaces, 2026-08-31, under AB#847.

| Thing | Value |
|---|---|
| Profile version | `1.0.0` (matches `profile.json` `profileVersion`) |
| Organization | `PolyphonyRequiem` |
| Project | `Twig-Reference-Sandbox` (`2c534971-1a18-4880-9cce-2ca1fb2c3cd6`) |
| Process | `Twig-Reference` (`a0afde20-50eb-4e30-b442-c9e7f13e752a`), inherited from Basic |
| Capture time | `2026-08-31T19:31:23.5195999+00:00` (frozen historical capture) |
| Team | `Twig-Reference-Sandbox Team` |
| Sprint iteration | `Twig-Reference-Sandbox\Sprint 1` (`9b7305e2-…`) |
| twig | `0.91.6-alpha.0.8` |

The committed fixtures freeze the capture timestamp, project/process IDs, fixture IDs,
and the exact ArtifactLink project/repository/ref identity used for surface 09.
`gate.sh` checks those fields against the committed evidence only; it proves internal
consistency of the bundle rather than current ADO truth.

## Layout

```
1.0.0/
├── README.md # this file
├── run-log.md # what was executed, in order, with digests
├── fixtures.json # the published fixture ids the scripts key on
├── capture-evidence.sh # re-capture the surfaces from live ADO
├── gate.sh # assert the ten pass criteria (step 15)
├── proposals/ # the proposal v1 files actually applied
└── evidence/ # 12 JSON artifacts; JSON is authoritative
```

## Running it

```bash
./gate.sh # offline: reads the committed evidence only
./capture-evidence.sh surfaces # online: re-captures 01-09 from live ADO
./capture-evidence.sh rank after # online: re-captures the rank snapshot
```

`gate.sh` reads nothing but `evidence/` and `fixtures.json`, so it runs in CI or offline.
It verifies the frozen bundle's internal consistency — including capture time,
project/process IDs, fixture IDs, and the exact artifact-link identity — rather than
current ADO truth. `capture-evidence.sh` needs `az` logged in against the org.

## The ten surfaces

| # | Surface | Evidence | Result |
|---|---|---|---|
| 01 | `Initiative` on the portfolio backlog | `01-initiative-backlog.json` | PASS |
| 02 | `Investigation` on the Requirements backlog | `02-investigation-work.json` | PASS |
| 03 | `Feature` on the Requirements backlog | `03-feature-work.json` | PASS |
| 04 | `Bug` on the Requirements backlog | `04-bug-work.json` | PASS |
| 05 | `Task` on the sprint board | `05-task-sprint.json` | PASS |
| 06 | Hierarchy renders as decomposition | `06-hierarchy-links.json` | PASS |
| 07 | Predecessor/successor renders as dependency | `07-predecessor-successor.json` | PASS |
| 08 | Related renders nondirectionally | `08-related-links.json` | PASS |
| 09 | Artifact link | `09-artifact-links.json` | PASS (see item 3 below) |
| 10 | Rank preserved across publish + link | `10-rank-{before,after}.json`, `10-rank-diff.txt` | PASS (diff empty) |

Fixture shape:

```
Initiative #857
├── Investigation #858 related ──> Feature #859
├── Feature #859 ArtifactLink ──> GBharness
│ ├── Task #861 predecessor of ──> Task #862
│ ├── Task #862
│ └── Task #863
└── Bug #860
```

The three Tasks are committed to `Sprint 1`; nothing else is. That is the sprint-entry
invariant, enforced structurally by the backlog behaviors rather than by convention.

### Screenshots

None were captured or committed. T2 §4.1 lists optional `.png` companions for the backlog
and sprint surfaces, and is explicit that "the `.json` proof is authoritative — the
harness gate reads JSON, not images". `gate.sh` never opens a PNG, so their absence does
not weaken the gate. Capturing them needs an authenticated browser session against the ADO
UI and is left to whoever wants the human-consumable view.

## Discrepancies found in T2 while executing

These were found by running the recipe and are corrected in the T2 note itself.

1. **Publish-step contradiction (T2 §2.6 vs §3 row 7).** §2.6 correctly states that an
inherited process has no publish step, while the §3 ordering table still carried a
human "Publish the process" row. There is no such transition; the row is removed.

2. **Surface-count contradiction (executive summary vs §4.3).** The summary said six
observation surfaces; §4.3 enumerates ten. Ten is correct and is what this bundle and
`gate.sh` implement.

3. **§4.2's plan document is not a real proposal file, and one of its ops cannot exist.**
The illustrative `plan.yaml` uses `ops` / `seed` / `alias` / `from` / `to` / `linkType`.
The native contract is proposal v1: top-level `version` + `workspace` + `operations`,
kinds `batch` / `add-link` / `remove-link` / `publish-seed` / `delete`, and a **closed**
relation set `parent | predecessor | successor | related`. Two consequences:
- `ArtifactLink` is **not expressible** as a proposal op. `twig proposal validate`
rejects it with `plan.invalid_relation`; the probe is kept at
`proposals/probe-artifact-link-REJECTED.json` as evidence. Surface 09 was captured via
direct ADO REST after that rejection, and the committed fixtures freeze the exact
project/repository/ref identity so the offline gate can assert it without credentials.
- Parent links must be expressed **child-side**. The native surface allows at most one
op per `workItemId` per proposal file, and §4.2's parent-side spelling puts three ops
on the Initiative in a single file.

4. **Rank row 10's capture recipe is unsatisfiable as written.** It says to capture
"`twig tree @FEAT` immediately after step 12's task publish, again after step 13's link
ops", and to compare the child order under `@FEAT`. Immediately after step 12 the Feature
has no children — they are created by step 13 — so the "before" snapshot is necessarily
empty and the comparison is vacuous. This run captures server-owned **backlog order**
for the portfolio and requirement backlogs before and after the link ops instead, which
is what "rank preserved across publish + link" actually means and is a non-vacuous test.

## Gating rule

Per T2 §5 this bundle is a hard gate for **publishing a new version of the reference
profile**, not a runtime gate and not a CI gate on every commit. A `profileVersion` bump
requires a bundle at the new version whose `gate.sh` exits 0. Twig core never reads
`docs/reference-process/harness/` at runtime; it validates the live process through
`IReferenceProfileProvider`. The harness gate is intentionally offline: it checks the
committed historical capture for internal consistency and never re-establishes current ADO
truth.

## Prior art

The step 2-9 human tailoring was performed under AB#727/AB#733 on 2026-08-27 and is *not*
re-executed here — it is re-verified live at the top of `run-log.md`. That run's own
evidence (18 REST responses covering the seven tailoring gates, plus the `VS403074` proof
that `Epic`/`Issue` creation is refused) lives outside the repo at
`~/.twig/harness/twig-reference/`.
Loading
Loading