Skip to content

Verify persona addressing teams in CI #1280

Closed
#1284
Closed
Verify persona addressing teams in CI#1280
#1284
Labels
dev-leadFor dev-lead agent pickupdocumentationDocumentation changesenhancementFeature requests

Description

@don-petry
don-petry
opened on Jul 17, 2026

Context

The persona standard (§4.1) requires a persona's address.handle to be an org team with specific live properties. Four places already assert that CI verifies this. Nothing does. See combsmith/.github#755 (finding 3).

The claim appears in:

  • standards/personas/persona.schema.json — "Verified by CI, not by this schema (needs the network)"
  • standards/persona-standards.md §4.1 — "verified by a separate CI step"
  • standards/personas/TEMPLATE/persona.yml — "CI verifies those live properties"
  • personas/qa-lead/persona.yml — "verified by CI, not by the schema"

Why it matters

Without the check, a persona can ship with:

  • privacy: secret → the team is unmentionable. The persona is silently dead; every mention does nothing and nobody learns why.
  • notification_setting: notifications_enabled → every mention pages the team's members. The handle exists to route a webhook, not to notify humans.
  • A team that does not exist at all → every mention silently no-ops.

All three fail silently, which is the failure mode this framework keeps producing. This is O(N) manual setup today with zero verification — it will not survive more personas.

Task

Add a CI check that, for every personas/<id>/persona.yml declaring an address block, verifies the live team:

  1. The team address.handle names exists.
  2. privacy == "closed" (secret teams cannot be mentioned).
  3. notification_setting == "notifications_disabled".
  4. The team slug equals id (already enforced hermetically by personas/validate-personas.py; do not duplicate — this check is about live state only).

Constraints (important)

  • Do NOT put this in personas/validate-personas.py. That validator is deliberately hermetic, and tests/test_validate_personas.bats is a hermetic suite. Its hermeticity is worth more than the check — that is precisely why this check was split out. Adding a network call there breaks the suite's contract.
  • Add it as a separate job in .github/workflows/lint.yml, alongside the existing validate-personas job (follow that job's shape).
  • Requires an org-scoped token to read teams (GH_PAT_WORKFLOWS is available as an org secret). gh api orgs/petry-projects/teams/<slug> returns privacy and notification_setting.
  • A persona without an address block must pass (address is optional unless the mention surface is enabled).
  • Fail with a clear diagnostic, never a traceback — match the ::error::persona manifest invalid: ... style already used by validate-personas.py.
  • Fail closed: an API error must NOT be read as "the team is fine". Conflating an error with a negative answer is the systemic bug this framework already has four instances of (feat: implement issue #737 — [Phase 1] Instrument auto-rebase: agentic-conflict-resolution rate + fan-out CI-run volume report #755). If the team cannot be read, fail the check.

Files that exist today (verified)

  • personas/validate-personas.py — the hermetic validator (do not add network here)
  • personas/qa-lead/persona.yml — the only persona; address.handle: petry-projects/qa-lead
  • .github/workflows/lint.yml — has the validate-personas job to model this on
  • tests/test_validate_personas.bats — hermetic suite, 20 tests

Acceptance criteria

  • CI fails when a manifest's address.handle names a team that does not exist.
  • CI fails when the team is privacy: secret.
  • CI fails when the team has notification_setting: notifications_enabled.
  • CI fails (does not pass) when the teams API errors.
  • CI passes for qa-lead as it stands today (team petry-projects/qa-lead is closed, notifications disabled, 0 members).
  • Personas with no address block pass.
  • tests/test_validate_personas.bats still passes and remains hermetic (no network).
  • New logic is shellcheck-clean if shell; if Python, it does not import into the hermetic validator's path.
  • The four docs above are updated only if the check's name/location differs from what they describe.

Ground truth for the happy path

$ gh api orgs/petry-projects/teams/qa-lead --jq '{slug,privacy,notification_setting,members_count}'
{"members_count":0,"notification_setting":"notifications_disabled","privacy":"closed","slug":"qa-lead"}

Activity

  1. don-petry
    added
    dev-leadFor dev-lead agent pickup
    on Jul 17, 2026
  2. github-actions
    added
    enhancementFeature requests
    documentationDocumentation changes
    on Jul 17, 2026

  3. github-actions commented on Jul 17, 2026

    @github-actions
    github-actionsBot
    on Jul 17, 2026 – with GitHub Actions
    Contributor

    Thanks for such a thorough write-up — the acceptance criteria, constraints, and ground-truth API output make this very actionable. One clarifying question: do you have a preference for the implementation language of the new job (shell script vs. Python)? The issue leaves both open, and the choice affects how the diagnostic output is structured and whether it fits naturally alongside the existing validate-personas.py tooling.

  4. don-petry commented on Jul 17, 2026

    @don-petry
    don-petry
    on Jul 17, 2026
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1280 — Verify persona addressing teams in CI

    Scope

    Add a separate, network-touching CI check (verify-persona-teams) that, for every personas/<id>/persona.yml declaring an address block, verifies the LIVE team named by address.handle: it exists, privacy == "closed", and notification_setting == "notifications_disabled" — failing closed on any API error. The hermetic validate-personas.py is left untouched.

    Implementation Checklist

    • scripts/verify-persona-teams.sh — new shell script (POSIX bash, set -euo pipefail, shellcheck-clean). Extracts address.handle slugs (python3+pyyaml), fetches gh api orgs/<org>/teams/<slug>, asserts exists + privacy=closed + notification_setting=notifications_disabled. Fails closed on any API/parse error with a ::error::persona team invalid: ... diagnostic. Personas with no address block pass.
    • .github/workflows/lint.yml — new verify-persona-teams job alongside validate-personas (installs pyyaml, runs the script with GH_TOKEN: secrets.GH_PAT_WORKFLOWS).
    • tests/verify_persona_teams.bats — hermetic unit/fixture suite (stubs gh, no network).
    • .github/workflows/lint.yml bats job list — add tests/verify_persona_teams.bats.
    • Docs: the four claim sites are name-agnostic; update only if wording diverges. (tests/test_validate_personas.bats already references the verify-persona-teams job by name.)

    Tests to Write

    • happy path: closed + notifications_disabled team → pass
    • privacy: secret → fail (diagnostic mentions privacy/closed)
    • notification_setting: notifications_enabled → fail
    • teams API error / nonexistent team → fail closed, not pass
    • team JSON missing fields → fail
    • persona with no address block → pass
    • no personas root → pass (no-op)
    • malformed handle → clear diagnostic, no traceback
    • tests/test_validate_personas.bats still passes and stays hermetic

    Test Command

    bats tests/verify_persona_teams.bats tests/test_validate_personas.bats + shellcheck --severity=warning -x scripts/verify-persona-teams.sh

  5. don-petry commented on Jul 17, 2026

    @don-petry
    don-petry
    on Jul 17, 2026
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • Added scripts/verify-persona-teams.sh — the LIVE, network-touching half of persona addressing validation.
    • Added a separate verify-persona-teams job to .github/workflows/lint.yml (alongside validate-personas, modeled on its shape).
    • Added hermetic suite tests/verify_persona_teams.bats and registered it in the bats job list.
    • Left personas/validate-personas.py and its hermetic suite untouched — no network added there.

    What the check does

    For every personas/<id>/persona.yml declaring an address block, it resolves address.handle to a team slug and calls gh api orgs/petry-projects/teams/<slug>, then asserts the LIVE team:

    1. exists (a 404 is an API error → fail),
    2. privacy == "closed" (secret teams are unmentionable),
    3. notification_setting == "notifications_disabled" (routes a webhook, never pages members).

    Fails closed: any API/parse error is a failure, never a pass (#755). Diagnostics use the ::error::persona team invalid: ... style, never a traceback. Personas with no address block pass. The slug==id invariant is not re-checked (already hermetic in validate-personas.py).

    Acceptance criteria

    • Fails when the team does not exist (404 → fail closed).
    • Fails when privacy: secret.
    • Fails when notification_setting: notifications_enabled.
    • Fails (does not pass) when the teams API errors.
    • Passes for qa-lead as it stands today — verified live end-to-end: personas/qa-lead/persona.yml → petry-projects/qa-lead (exists, closed, notifications disabled).
    • Personas with no address block pass.
    • tests/test_validate_personas.bats still passes and stays hermetic (no network).
    • Shellcheck-clean (--severity=warning -x); actionlint-clean; .dev-lead/scripts/dev-lead-lint.sh passes.
    • Docs: the four claim sites are name-agnostic (none names a specific job); tests/test_validate_personas.bats already references the verify-persona-teams job by that exact name — so no doc edits were required.

    Test Results

    ok 29 validate-personas accepts distinct handles across personas
ok 30 validate-personas ignores personas with no address block
ok 31 validate-personas reports a bare handle with no org prefix
ok 32 validate-personas reports a non-string handle
ok 33 validate-personas reports a malformed alias
ok 34 validate-personas reports a non-list aliases

    Files Changed

    • scripts/verify-persona-teams.sh (new): the live team check (bash + python3/PyYAML + jq).
    • .github/workflows/lint.yml: new verify-persona-teams job (token GH_PAT_WORKFLOWS) + tests/verify_persona_teams.bats added to the bats list.
    • tests/verify_persona_teams.bats (new): 14 hermetic tests (gh stubbed, no network).

    Notes

    • The job passes GH_TOKEN: secrets.GH_PAT_WORKFLOWS because github.token cannot read org teams. On a token-less context the read fails and the job fails closed — the intended behavior. The Lint workflow's personas/** path filter scopes the job to persona changes.
    • The network call is isolated behind vpt_team_api/gh, keeping the suite hermetic (mirrors the validate-personas split).
  6. don-petry
    added a commit that references this issue on Jul 17, 2026

    feat: implement issue #1280 — Verify persona addressing teams in CI (#…

    e32a848
  7. don-petry
    closed this as completedin #1284on Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupdocumentationDocumentation changesenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions