Cross-layer validation: close the workflow ↔ conductor ↔ polyphony-verb contract gap

[PolyphonyRequiem]

Cross-layer validation: workflow ↔ conductor ↔ polyphony-verb

Problem

Three independent layers participate in every workflow step, and nothing
checks that they agree until a dogfood run actually exercises the step:

1. Workflow YAML (.conductor/registry/workflows/*.yaml) — references
   verb output fields like {{ derive_ancestor_chain.output.parent_item_id | default(0) }}
   and calls Jinja filters/functions. conductor validate checks schema
   shape only. Hand-written lint-*.ps1 greps in
   .conductor/registry/tests/ check spelling but cannot resolve a
   reference back to its producer.
2. Conductor Jinja runtime (conductor/src/conductor/executor/template.py) —
   custom filter set (_default_filter accepts only 2 positional args),
   strict_undefined on. Lints can require Jinja constructs that
   conductor does not actually expose.
3. Polyphony verb output — C# DTOs serialized via
   PolyphonyJsonContext (src/Polyphony/PolyphonyJsonContext.cs:172,
   DefaultIgnoreCondition = WhenWritingNull). Nullable fields are
   silently elided from the JSON wire shape, so a schema-present field
   can be a runtime-missing key.

Each layer is well-tested in isolation. The contract between them is
checked only by execution.

Evidence — recent dogfood-discovered bugs

Bug	PR(s)	Class
#6	#168	Field-name drift: YAML read type_loader.output.type_name; verb emits type.
#7	#169	Workflow + lint required severities_at_or_above(...) Jinja function; conductor never registered it. Fix: precompute as a verb output field.
#8	#170, #171	derive-ancestor-chain.parent_item_id (int?) elided by WhenWritingNull on root path; strict_undefined raised before default(0) could fire. Then default(0, true) crashed because conductor's default filter is 2-arg.
#159 family	#159, tracked in #165	polyphony CLI exits 0 on unrecognized verbs/args, so YAML→CLI signature drift is invisible.

Each one only surfaced when execution reached the offending step. All four
were mechanically detectable at lint time given the right cross-layer
metadata.

The catalog at docs/polyphony-state-effects-catalog.md § "Open questions
/ gaps" already documents these as a class but offers no fix track.

Proposed fix shape — four prongs

Tracked as child issues. Each is independently shippable.

1. Verb output schema registry — [Validation gap] Verb output schema registry generated from PolyphonyJsonContext #173. Generated from
   PolyphonyJsonContext source. Per-field WhenWritingNull vs Never
   behaviour encoded. Published as verb-output-schemas.json artifact
   consumable by lint and humans. Wire shape is the load-bearing detail
   (bug [PG-1] Core Infrastructure and Routing Types #8).
2. YAML Jinja-reference resolver lint — [Validation gap] Workflow YAML Jinja-reference resolver lint #175. For every
   {{ X.output.field.path }} in workflow YAML, look up X's verb in
   the registry, walk the field path, fail lint if missing or if the
   field can be omitted at runtime without a default() guard. Catches
   bug Feature/2582 pg 3 #6 (name drift) and bug [PG-1] Core Infrastructure and Routing Types #8 (omit-when-null without guard).
   Depends on [Validation gap] Verb output schema registry generated from PolyphonyJsonContext #173.
3. Conductor Jinja inventory pin — [Validation gap] Conductor Jinja inventory pin and lint enforcement #174. Generated manifest of
   filters and globals conductor's TemplateRenderer actually exposes,
   with signatures. Lint refuses any Jinja construct outside that set.
   Maintained by a small probe against the pinned conductor version.
   Catches bug Feature/2582 pg 4 #7 (nonexistent function) and bug [PG-1] Core Infrastructure and Routing Types #8 fixup (wrong-arity
   filter call).
4. Polyphony CLI strict-mode — polyphony CLI exits 0 on unrecognized args (silent-failure root cause) #165 (pre-existing, adopted as the
   prong-4 child). Exit non-zero on unrecognized
   verbs/subcommands/options. No upstream deps; ship first.

What this is NOT

• Not changing how workflows are authored. Same Jinja, same YAML.
• Not introducing a DSL. The registry is generated, not hand-maintained.
• Not blocking on conductor changes. Prongs 1, 2, 4 are owned by
  polyphony. Prong 3 needs only a probe script run against the conductor
  version polyphony already targets (min_polyphony_version in workflow
  metadata; no formal conductor SHA pin today — that's part of [Validation gap] Conductor Jinja inventory pin and lint enforcement #174's
  design).

Acceptance

• All four child issues closed: polyphony CLI exits 0 on unrecognized args (silent-failure root cause) #165, [Validation gap] Verb output schema registry generated from PolyphonyJsonContext #173, [Validation gap] Conductor Jinja inventory pin and lint enforcement #174, [Validation gap] Workflow YAML Jinja-reference resolver lint #175.
• Each of bugs Feature/2582 pg 3 #6 / Feature/2582 pg 4 #7 / [PG-1] Core Infrastructure and Routing Types #8 has a regression test that would have caught
  it at lint time, executed in CI.
• Catalog § "Open questions / gaps" entries for these bugs collapse into
  links to the shipped lint.

Children

• [Validation gap] Verb output schema registry generated from PolyphonyJsonContext #173 — Verb output schema registry (prong 1)
• [Validation gap] Workflow YAML Jinja-reference resolver lint #175 — Workflow YAML Jinja-reference resolver lint (prong 2, depends on [Validation gap] Verb output schema registry generated from PolyphonyJsonContext #173)
• [Validation gap] Conductor Jinja inventory pin and lint enforcement #174 — Conductor Jinja inventory pin and lint enforcement (prong 3)
• polyphony CLI exits 0 on unrecognized args (silent-failure root cause) #165 — Polyphony CLI strict-mode (prong 4, pre-existing; ship first)

[PolyphonyRequiem]

Bug catalog update — 2026-05-07 dogfood iter 8/8b:

Three more bugs surfaced from the same disease class (apex pipeline / canonical-builder drift):

• Bug #10: worktree-manager.ps1 spawn not branch-idempotent (stale sdlc/apex/{id} blocks resume) #177 — worktree-manager.ps1 spawn not branch-idempotent
• Bug #11: apex-item-dispatch terminal_* nodes missing canonical output schema (renegotiation_pending et al) #178 — apex-item-dispatch terminal_* nodes missing canonical output schema
• Bug #12: .polyphony/run.yaml never committed/pushed to feature branch (structural manifest-lifecycle gap) #179 — .polyphony/run.yaml never committed/pushed to feature branch (structural manifest-lifecycle gap)

#179 is the most interesting — it's a design-spec gap, not just an authoring slip. The branch-model ADR names the invariant ("manifest lives on feature branch") but doesn't assign ownership (who commits, when, push semantics, mutation lifecycle).

This brings the running tally of cross-layer-validation bugs caught during dogfood to: #9 (apex- prefix drift, fixed PR #176), #177, #178, #179.

The pattern across all four: YAML/PowerShell hand-rolls logic that diverges from (or is absent in) the C# canonical layer. Resolver lint (#175) is the highest-leverage prevention; verb output schema registry (#173) catches #178's class.

[PolyphonyRequiem]

Closing this epic. Two of three prongs shipped; prong 3 (#174) is deferred.

Shipped

• Prong 1 ΓÇö [Validation gap] Verb output schema registry generated from PolyphonyJsonContext #173 verb output schema registry (PR feat(schema): build-time verb output schema registry (#173) #184). Source-generated catalog of every CLI verb's result type, walked transitively across DTO graphs. 86 verbs, 122 types, 26 tests.
• Prong 2 ΓÇö [Validation gap] Workflow YAML Jinja-reference resolver lint #175 Jinja resolver lint (PR feat(lint): jinja resolver lint (#175) #186). Statically resolves {{ <step>.output.<path> }} in workflow YAMLs against the registry. 5 diagnostic codes, 26 Pester tests, surfaced 4 real bugs in plan-level.yaml on first real-corpus pass (filed as Bug: plan-level.yaml references undefined fields on merge_plan_pr / merge_plan_pr_ado output #185).
• Prong 4 ΓÇö polyphony CLI exits 0 on unrecognized args (silent-failure root cause) #165 polyphony CLI strict-mode. Pre-existing, owned separately; status tracked on its own issue.

Together these close the workflow YAML → polyphony verb output half of the contract.

Deferred ΓÇö #174 (conductor Jinja inventory pin)

Stays open as a tracked gap, but explicitly punted from this epic. The remaining failure mode it would catch ΓÇö calls to nonexistent or wrong-arity Jinja filters/globals/tests (bugs #7 and #8 fixup) ΓÇö is real but appears at the rate of roughly two incidents per several weeks of dogfooding. The probe + lint scope is small; the load-bearing piece is deciding where polyphony pins a conductor version, which today is informal.

The pragmatic call: re-open #174 actively when a third bug in this class surfaces. By then we will have lived under the registry + resolver lint for a while, and we will have a better signal on conductor-version churn cadence to inform the pin design.

What this epic delivered

• Closed the workflow ΓåÆ verb-output contract gap (the bigger half).
• Built the verb output schema registry, which is independently useful (debug, doc generation, future contract checks).
• Hardened against the field-name and omit-when-null bug classes (Feature/2582 pg 3 #6 and [PG-1] Core Infrastructure and Routing Types #8) that motivated the epic.
• Surfaced a fresh bug (Bug: plan-level.yaml references undefined fields on merge_plan_pr / merge_plan_pr_ado output #185) on first lint pass ΓÇö concrete proof the lint is doing work.

The conductor-Jinja inventory remains a known gap; revisit when warranted.