Skip to content

feat(tui): capability invocation telemetry on Performance screen - #15

Merged
flg77 merged 1 commit into
mainfrom
feat/capability-invocation-telemetry
May 1, 2026
Merged

feat(tui): capability invocation telemetry on Performance screen#15
flg77 merged 1 commit into
mainfrom
feat/capability-invocation-telemetry

Conversation

@flg77

@flg77 flg77 commented May 1, 2026

Copy link
Copy Markdown
Owner

Summary

Wires the `TASK_COMPLETE.invocations` field PR-B added all the way to the operator's eye: the NATSObserver folds each entry into the CollectiveSnapshot, and the Performance screen renders two new panels — running per-(kind, target) totals + a recent-failures tail — so operators see which skills + MCP tools are firing, at what success rate, and what the latest failure was.

Architecture

```
Agent ── publishes ─▶ TASK_COMPLETE { …, invocations: [{kind,target,ok,error}, …] }


NATSObserver._route_task_complete


CollectiveSnapshot.record_invocation
├── capability_stats[k]: {total, ok, last_error, last_seen_ts}
└── invocation_log: FIFO tail (cap 50)


PerformanceScreen.watch_snapshot
├── _render_capability_invocations → DataTable (sorted by total desc)
└── _render_capability_failures → Static (last 10 failures)
```

Files

File Change
`acc/tui/models.py` New `CapabilityInvocationStats` dataclass + `CollectiveSnapshot.capability_stats` dict + `invocation_log` FIFO + `record_invocation()` helper.
`acc/tui/client.py` `NATSObserver._route_task_complete` now iterates `data["invocations"]` and folds each entry. Pre-PR-B agents (no `invocations` field) are a no-op.
`acc/tui/screens/performance.py` Two new panels in the right column: `CAPABILITY INVOCATIONS` DataTable + `RECENT FAILURES` Static.
`acc/tui/help/performance.md` Two new §sections describing the panels + colour legend.
`tests/test_capability_telemetry.py` (new) 13 cases.

Visuals

```
CAPABILITY INVOCATIONS (skill / MCP tool)
┌──────────┬───────────────────────┬───────┬──────┬──────────────────────────┐
│ Kind │ Target │ Total │ OK% │ Last error │
├──────────┼───────────────────────┼───────┼──────┼──────────────────────────┤
│ mcp │ echo_server.echo │ 142 │ 100% │ — │
│ skill │ echo │ 87 │ 98% │ schema mismatch │
│ mcp │ fs.read │ 23 │ 78% │ A-018 blocked │
└──────────┴───────────────────────┴───────┴──────┴──────────────────────────┘

RECENT FAILURES (latest 10)
14:32:18 mcp:fs.read coding-aabb
A-018 blocked tool 'fs.read'@'fs_server': mcp_server 'fs_server' not in role.allowed_mcps
14:31:55 skill:echo coding-aabb
schema mismatch: 'text' must be string
```

Tests (13 new, all green)

Group Cases
`CapabilityInvocationStats` initial state, fail + ok_rate
`CollectiveSnapshot.record_invocation` first call, accumulation, kind:target keying, malformed drops, FIFO cap
`NATSObserver._route_task_complete` folds invocations, missing field is no-op, non-list field is safe
`PerformanceScreen` Pilot sort-by-total-desc, failures-only panel, empty-state guidance

Combined regression: 166 existing tests pass unchanged (config + role-store + guardrails + compliance + observer + prompt-channel + ecosystem + file-picker; TestEd25519Validation deselected for lighthouse OpenSSL platform limit).

Test plan

  • `pytest tests/test_capability_telemetry.py -v --no-cov` → 13 passed
  • `pytest tests/test_config.py tests/test_role_store.py tests/test_guardrails.py tests/test_compliance.py tests/test_observer_task_listener.py tests/test_prompt_channel.py tests/test_ecosystem_screen_pilot.py tests/test_file_picker_pilot.py --deselect tests/test_role_store.py::TestEd25519Validation --no-cov -q` → 166 passed
  • Manual: `acc-tui` → press `5` → Performance. Submit a few tasks via the prompt pane (or have an agent emit a few `[SKILL: echo {...}]` markers) → confirm the CAPABILITY INVOCATIONS table populates with rows, OK% colours match the 95/80 thresholds.

Out of scope (planned follow-ups)

  • TASK_PROGRESS streaming for prompt channels.
  • Telegram + WhatsApp PromptChannel adapters.
  • Real echo MCP server image + compose entry.

PR-B added an ``invocations`` field to the TASK_COMPLETE payload
(``[{"kind", "target", "ok", "error"}, ...]``) but nothing was
consuming it.  This PR wires the field end-to-end: the NATSObserver
folds each entry into the CollectiveSnapshot, and the Performance
screen renders two new panels — running per-(kind, target) totals
plus a recent-failures tail — so operators see at a glance which
skills + MCP tools are firing, at what success rate, and (when the
rate drops) what the most recent failure was.

acc/tui/models.py
  + CapabilityInvocationStats dataclass — kind, target, total, ok,
    last_error, last_seen_ts.  Properties .fail and .ok_rate.
    Unfired tools report ok_rate=1.0 so the panel doesn't accuse a
    never-invoked tool of failing.
  + CollectiveSnapshot.capability_stats: dict[str, ...] keyed by
    f"{kind}:{target}" so a skill named "echo" and an MCP tool also
    named "echo" don't collide.
  + CollectiveSnapshot.invocation_log: FIFO tail capped at
    _MAX_INVOCATION_LOG (50).
  + record_invocation(invocation, *, agent_id, task_id, ts) — folds
    one entry into capability_stats + appends to invocation_log.
    Drops malformed entries silently (unknown kind, missing target).

acc/tui/client.py
  NATSObserver._route_task_complete now iterates data["invocations"]
  and calls snapshot.record_invocation for each well-formed entry.
  Pre-PR-B agents that omit the field are no-ops (no crash, no stats).
  Non-list invocations field is also skipped safely.

acc/tui/screens/performance.py
  + CAPABILITY INVOCATIONS DataTable — Kind / Target / Total / OK% /
    Last error.  Sorted by total desc so the busiest tools surface.
    Empty state shows a single grey hint pointing at
    docs/howto-skills.md.  OK% colour-coded:
      ≥ 95 %  green
      ≥ 80 %  yellow
      else    red
    Kind cell colour-coded cyan (skill) / magenta (mcp).
  + RECENT FAILURES Static — last 10 failures from invocation_log
    (successes filtered out).  Format: "ts  kind:target  agent_id"
    + indented error line.  Empty state shows a grey hint.
  Both panels live in the existing right-column Vertical alongside
  TOKEN BUDGET / LATENCY PERCENTILES.

acc/tui/help/performance.md
  Two new §sections describing the panels, including the colour
  legend + sort order + empty-state behaviour.

Tests (tests/test_capability_telemetry.py — NEW, 13 cases all green):
  CapabilityInvocationStats (2):
    initial state clean; fail + ok_rate track correctly.
  CollectiveSnapshot.record_invocation (5):
    creates stats on first call,
    accumulates across calls + tracks last_error,
    keys kind:target independently (skill echo vs mcp echo),
    drops malformed entries silently,
    invocation_log enforces FIFO cap at 50 (60-call overflow test).
  NATSObserver._route_task_complete (3):
    folds invocations into snapshot end-to-end,
    handles missing invocations field (pre-PR-B compat),
    handles non-list invocations safely.
  PerformanceScreen Pilot (3):
    capability table sorted by total desc,
    failures panel lists only failures (success row excluded),
    empty snapshot renders the guidance row.

Combined regression: 166 existing tests pass unchanged
(test_config, test_role_store, test_guardrails, test_compliance,
test_observer_task_listener, test_prompt_channel,
test_ecosystem_screen_pilot, test_file_picker_pilot;
TestEd25519Validation deselected for the lighthouse OpenSSL platform
limit, as in earlier PRs).

Out of scope (planned follow-ups):
* TASK_PROGRESS streaming for the prompt channels
  (TUIPromptChannel.supports_streaming() flips True).
* Telegram + WhatsApp PromptChannel adapters.
* Real echo MCP server image + compose entry.
@flg77
flg77 merged commit 83b8198 into main May 1, 2026
@flg77
flg77 deleted the feat/capability-invocation-telemetry branch May 1, 2026 11:05
flg77 added a commit that referenced this pull request May 1, 2026
…ier B) (#22)

Tier B of the coding_agent test plan (Tier A — schema invariants —
landed in #21).  Verifies that the canonical TUI surfaces consume
the ``coding_agent`` role correctly:

* Ecosystem ROLE LIBRARY contains the row.
* ROLE DETAIL panel renders the role's purpose + persona on selection.
* SKILLS / MCP SERVERS tables list echo + echo_server.
* "Schedule infusion → Nucleus" button dispatches RolePreloadMessage
  with role_name='coding_agent'.
* Prompt pane (screen 7) defaults the target_role Select to
  coding_agent; Send publishes a TASK_ASSIGN with target_role +
  optional target_agent_id pinning.
* Performance screen (PR #15) renders the CAPABILITY INVOCATIONS
  table + RECENT FAILURES panel from synthetic TASK_COMPLETEs
  sourced from a coding_agent-* agent_id, including A-017 block
  reasoning.

tests/test_coding_agent_tui_pilot.py (NEW, 10 cases all green):

  Ecosystem (5):
    test_ecosystem_role_library_row_for_coding_agent
      → ROLE LIBRARY contains the coding_agent row keyed by name.
    test_ecosystem_role_detail_renders_coding_agent_seed
      → on RowSelected, ROLE DETAIL Static carries the role.yaml
        purpose phrase verbatim + the "analytical" persona.
    test_ecosystem_skills_table_shows_echo_for_coding_agent_role
    test_ecosystem_mcps_table_shows_echo_server_for_coding_agent_role
      → both confirm PR-A's path-resolution fix loads the repo's
        skills/ + mcps/ correctly even when the test runner's cwd
        differs from the repo root.
    test_schedule_infusion_button_dispatches_role_preload_for_coding_agent
      → Send button posts RolePreloadMessage(role_name='coding_agent')
        the App routes to InfuseScreen.preload_from_role.

  Prompt pane (3):
    test_prompt_pane_target_role_defaults_to_coding_agent
      → guards a regression where the default Select value drifts.
    test_prompt_send_routes_task_assign_to_coding_agent
      → Send publishes TASK_ASSIGN with target_role='coding_agent',
        no target_agent_id (broadcast-by-role behaviour).
    test_prompt_send_with_target_agent_id_pins_to_specific_coding_agent
      → operator-supplied target_agent_id makes it onto the wire so
        PR-B's filter on the agent side only lets the named agent
        process the task.

  Performance screen (2):
    test_performance_telemetry_records_coding_agent_skill_invocation
      → coding_agent emits a skill:echo invocation in TASK_COMPLETE
        → CAPABILITY INVOCATIONS table renders one row keyed
        ``skill:echo``.
    test_performance_failures_panel_renders_a_017_block_from_coding_agent
      → coding_agent attempts shell.exec → Cat-A A-017 blocks →
        RECENT FAILURES panel shows shell.exec + A-017 + the
        coding_agent agent_id (truncated).

Implementation patterns reused from earlier PRs:
  * ``_capture_static_updates`` — PR-A trick to read Static.update
    calls across Textual versions.
  * ``_StubObserver`` (channel-shape) — PR-B harness for prompt-pane
    pilot tests.
  * Synthetic TASK_COMPLETE delivery via ``observer._route_task_complete``
    — PR #15 telemetry test pattern.
  * Synthetic ``DataTable.RowSelected`` event — PR-A pattern, more
    deterministic than ``pilot.press("enter")``.

Combined regression: 238 unit tests pass on Windows
(test_coding_agent_role + test_coding_agent_tui_pilot +
test_task_progress_emit + test_task_progress_streaming +
test_prompt_screen_pilot + test_prompt_channel +
test_observer_task_listener + test_redis_compat +
test_capability_telemetry + test_ecosystem_screen_pilot +
test_file_picker_pilot + test_config + test_role_store +
test_guardrails + test_compliance, with TestEd25519Validation
deselected for the lighthouse OpenSSL platform limit).

Out of scope (next two follow-up PRs):
* Tier C — Live container integration via the coding-split profile.
  Opt-in via ACC_LIVE_TESTS=1 + a running stack so CI doesn't hang
  on a missing NATS.
* Tier D — Negative paths: missing role dir → graceful EcosystemScreen
  degradation, target_agent_id mismatch → drop, HIGH-risk skill
  blocked by A-017 ceiling.
flg77 added a commit that referenced this pull request Jun 2, 2026
…9) (#15)

Two focused follow-ups to v0.3.48:

* **webgui startup warning** — `resolve_auth_config` now logs a loud
  WARNING when `ACC_WEBGUI_AUTH_MODE=htpasswd` AND the configured
  `ACC_WEBGUI_HTPASSWD_PATH` is missing / unreadable / unset.
  Triggered by today's lighthouse smoke: the operator's host-path
  env (`/home/flg/...`) was propagated into the container where the
  file was mounted at `/app/...`, every login silently 401'd, and the
  only signal in `podman logs` was uvicorn's normal startup. Now the
  operator sees the mismatch in the first log line after the auth
  mode is set.
* **ssh_exec skill** — HIGH-risk sibling to shell_exec for verifying
  remote outcomes from inside an agent. `requires_actions: [execute_ssh]`,
  argv-only (shlex.join'd into the remote command), key-auth only
  (`BatchMode=yes`), host allowlist via `ACC_SSH_HOST_ALLOWLIST`
  (localhost always allowed), default `accept-new` StrictHostKeyChecking,
  60s/600s timeout, output cap. Flipped onto the same 9-role
  engineering family that already has shell_exec.

Tests: +14 across 2 new files; full sweep 2561 passed, 38 skipped,
0 failed.

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
flg77 added a commit that referenced this pull request Jun 29, 2026
…cryptography (#141)

Resolves the Python (pip) Dependabot alerts on flg77/acc-spearhead. Most were a
stale uv.lock (last generated 2026-06-14) whose pins lagged the already-permissive
pyproject constraints; one needed a constraint widening.

Lockfile refresh (constraints already allowed the fix — `uv lock --upgrade-package`):
  aiohttp   3.14.0 → 3.14.1   (#15-22, 8 alerts)
  pyjwt     2.12.1 → 2.13.0   (#9-13, 5 alerts)
  starlette 1.0.1  → 1.3.1    (#24-27, 4 alerts; transitive via fastapi)
  msgpack   1.1.2  → 1.2.1    (#37, high)
  joserfc   1.6.5  → 1.7.2    (#39; transitive via authlib)

Constraint widening (fix was outside the pin):
  cryptography 46.0.7 → 48.0.1 (#14/#23, two HIGH) — pyproject `<47`→`>=48.0.1,<49`.
  The Ed25519 arbiter sign/verify API is stable across 46→48; operator "pin and
  bump" one major at a time.

The lock also catches up on the speech/turbovec optional-deps added since 06-14
(faster-whisper/piper-tts/ctranslate2/onnxruntime/av/turbovec) — no security
content, just lock/pyproject reconciliation. No downgrades.

Not fixed here (separate handling):
  transformers (#1) — vulnerable `Trainer` not in our execution path; ST<5.0 blocks
    the 5.0.0rc3 fix → dismissed on GitHub with that reason (intent already in
    pyproject).
  torch (#7, low) — no patched release exists → dismissed "no fix available".
  npm console-plugin (7 alerts) — need npm/node (absent here) → fleet hand-off.

Verified with the bumped libs installed: signatures/spiffe 153 ✓, a2a/messenger/
slack/webgui/redis 168 ✓ (321 total, 0 failures).

Co-authored-by: flg <flg@acc1.ic3net.internal>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant