Repository navigation
feat(reporting): record what sessions and workflow steps delivered, and count it per run mode - #1471
Conversation
The description includes sufficient built real-CLI verification. No testing-record blocker; I did not run, build, or install PR code. |
|
Both fixed in d0e9d80.
|
|
No new findings in the specified PR diff. Both earlier findings are resolved:
The description documents sufficient built real-CLI verification. Deferring other providers and agents to CI is acceptable; no testing-record blocker. I did not run, build, or install PR code. |
SaulMoro
left a comment
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
@hz89Dev Thanks. We compared the built CLI with main (84fb146):
- Local records. A Stop records
gitHeadandgitBranch, a learning getssessionin its frontmatter, and a step's pass gets the learning itscontributerecorded. - What the team sees. The stats file gets only counts, under
outcomes. A grep found no sha, session id, branch or link there, and a secondpullchanged nothing. - Readers.
digestshows the new table. The dashboard's/api/outcomesand run-detail views return the outcomes. - 0.28.0 on the same team repo. It reads this build's stats and run records, and keeps
outcomeswhen it rewrites the stats file. - CI review. d0e9d80 fixes both of its findings: the first commits in a repo with no commits yet, and the PR lookup needing
origin/HEAD.
One change before this can be approved:
- [P1 blocking] The first report after an upgrade counts every past session as "delivered nothing". We used a local event log written by
main, with 3 finished sessions: two committed, and one of those also contributed a learning. The firstteamai pullon this build wroteoutcomes.attended.none.sessions: 3, anddigestprintedattended none 3. The log is only compacted above 5,000 lines, so most members will hit this. The counters are cumulative, so nothing corrects them later. Fix: when there is noreported-outcomes.jsonyet, mark the sessions already finished as reported without counting them.
Non-blocking:
- [P2 non-blocking] A PR merged within 6 hours of the session is never counted; the body lists this as a known limit.
- [P3 nit] Two concurrent sessions in one checkout each get the other's commits.
- [P3 nit]
reported-outcomes.jsongrows without bound, as the existing baselines do. - [P3 nit] The body says "pull / push", but only
pullsends the report. - [P3 nit] A git wrapper that leaves a child holding the pipe after the 2-second abort loses the whole Stop event.
- [P3 nit]
contributecan append to an attempt that has already settled, where nothing reads it. - [P3 nit] After a downgrade, 0.28.0's dashboard shows phantom live sessions.
Besides the counts, three new things reach the team repo, as #1387 describes:
- a step's commit shas in the run's end record, which carried no shas before;
- the step's learning ids, in the same record;
- the session id, in the learning's frontmatter.
Overlap: textual conflicts with #1466 and #1379, no duplicates.
Verified at d0e9d80 (the hook and mixed-release runs at 5d112a8, which d0e9d80 doesn't touch): tsc and lint pass, the e2e files (116 tests) and self-mode-worktrees-808 pass, and the new tests fail on main. CI is green. Not run here: a real GitHub or GitLab PR lookup, other agents, and the dashboard in a browser.
@jeff-r2026 This adds data the team repo receives (#1387), so it's yours to decide.
d0e9d80 to
8535652
Compare
|
The blocking finding is fixed in 3c7bbb9, two of the nits in 3c7bbb9 and 8535652, and the rest are left as they are, with reasons below.
Not changed:
|
Both earlier findings are resolved: first commits are detected, and PR lookup no longer requires The description includes sufficient built real-CLI verification; no testing-record blocker. Deferring other providers and agents to CI is acceptable. I did not run, build, or install PR code. |
|
Fixed in f3dbe8f. The kept lookup now names its branch. A session that last stopped on another branch is asked about that one, and only the answer for that branch counts, so the earlier branch's PR is left out even when the later branch was never pushed or its lookup fails. Covered by "counts the pull request of the branch the session last stopped on, not one a report found on an earlier branch": a report finds PR 41 on |
|
No new findings in the specified PR diff. All three earlier findings are resolved:
The description documents sufficient built real-CLI verification; no testing-record blocker. Deferring additional providers and agents to CI is acceptable. I did not run, build, or install PR code. |
SaulMoro
left a comment
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
Re-review at f3dbe8f, after the rebase onto main. Nothing blocking is left on our side. Whether these new records belong in the team repo (#1387) is still the maintainer's call, as in our earlier comment.
Earlier findings
| Finding | Status |
|---|---|
[P1] The first report after an upgrade counted every past session under none |
Resolved in 3c7bbb9. Built CLI: main recorded 3 sessions and reported. This build's pull then reported no outcomes and marked the 3 as reported. A new session with a commit counted under commit, and a second pull changed nothing. The new test fails on the old head's source. |
| [P2] A PR merged within 6 h was never counted | Resolved in 3c7bbb9. The body lists the remaining limit (opened and merged between two reports). |
| [P3] Body said "pull / push" | Resolved. |
| [P3] A git wrapper holding the pipe lost the Stop | Resolved in 8535652. |
| Four other nits | Declined with reasons. We accept them. |
The CI review's branch-switch finding is fixed in f3dbe8f: the kept lookup records its branch, and only the last-stopped branch counts.
New, non-blocking
- [P3 nit] A still-going session with pushed commits and no PR yet is looked up again on every report, because a
noneanswer is not kept (src/session-outcomes.ts:243-254). It costs API calls and some latency on a manualpull. Nothing fails. - [P3 nit] An unreadable
reported-outcomes.jsonreads as a first report, so finished sessions are marked as reported without being counted once. Before, they were double-counted, so this is no worse. - [P3 nit] On a fresh install, sessions that finish before the scope's first report are skipped. That needs a failed or offline first report for 6 h or more. The body says so.
Verification at f3dbe8f
npm ci,npm run build,npx tsc --noEmit,npm run lintexit 0.- The PR's 10 unit files: 604 of 606 passed. The two failures are
workflow-local-driverlease-timeout tests from main that fail only under heavy machine load; they pass on rerun. - 7 related e2e files: 35 of 35 passed. The full e2e suite was not rerun; the earlier review covered it, and these files reach the new commits.
- CI is green at this head, and the body keeps its real-CLI record.
Not run here: a real GitHub or GitLab PR lookup, agents other than Claude, the dashboard in a browser.
SaulMoro
left a comment
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
Approving at f3dbe8f. The review is in our comment above: the first-report-after-upgrade finding is fixed and checked with the built CLI across a release upgrade, CI is green, and the body keeps its real-CLI record. The three nits there (repeat lookups for a still-open session, an unreadable reported-outcomes.json, an offline first report on a fresh install) can follow later.
#1466 just merged and this branch now conflicts with main. Could you rebase onto main? We will check the rebase and merge once CI is green at the new head. |
contribute writes session: { platform, sessionId } into the learning's
frontmatter before queuing it, records the learning as an outcome of
that session in the local dashboard event log, and inside a workflow
attempt appends it to the attempt's outcome file. A learning's id is
its path under learnings/ without .md.
Refs Tencent#1387
The engine reads the attempt's outcome file when it settles the attempt, for every driver, and keeps its entries on the pass. The end record gives each pass its outcomes: the commits on its branch, the pull request delivered for that step and pass, and what its own commands reported. The run keeps prs; its union is derived from the steps. Refs Tencent#1387
…s apart The Stop event records gitHead, so a session's commit range is its start's gitAtStart.head to its last Stop's gitHead. Session outcome lines stay out of what every existing reader of the event log reads, and compaction keeps them for 30 days, verbatim, after their session's own events are dropped. Refs Tencent#1387
The usage report adds each of the member's sessions to stats/<user>.yaml under outcomes.<runMode | notRecorded>.<pr | commit | learning | none> once it has been idle for 6 hours: sessions, tokens and interventions. A session is counted once, flagged in reported-outcomes.json beside the other baselines, which also keeps what its events said after compaction drops them. Commits are the member's own in the session's start..last-Stop range, dated after it started. A pull request is the open one of the checkout's branch holding one of them, looked up once and cached in outcome-prs.json. Ids, shas and URLs stay in the local event log; the report carries counts only. Refs Tencent#1387
The digest adds up the team's outcome rows and shows, per run mode and outcome, the sessions with their tokens and interventions per session. A session in two rows is said to be, so the rows are not read as a sum. Refs Tencent#1387
A run's steps table gains an outcomes column: the pass's commits, pull requests and learnings, from the local run or the published record. An outcome the page cannot read is left out, and a pull request links only on https. Refs Tencent#1387
Sessions are rebuilt with their outcome events folded in, so the session table and a session's detail show the learnings recorded for it, and the commits and pull requests the usage report found once it finished. Refs Tencent#1387
The overview gains a panel built from the sessions the usage report counted in the last 30 days: the share that delivered a commit, pull request or learning, the rows per run mode and outcome with tokens and interventions per session, and the latest sessions with their commits, pull request links and learnings. A workspace shows the sessions its own report counted. Refs Tencent#1387
…edge guides Clarify contribute --session-id: the session is recorded on the learning and counted as that session's outcome. Refs Tencent#1387
…written A learning recorded inside a workflow attempt is appended to the attempt's outcome file. A write error there no longer fails the contribution after the learning is queued; it warns, and the session still counts as contributed. Refs Tencent#1387
…topped on A Stop now records the branch beside the commit it saw. The report looks up the pull request of that branch, so a member who switched branches after opening it, before the session was finished, still has it counted. Also stub the new Stop helper in the hook handler test's collector mock. Refs Tencent#1387
Reuse CommitIdSchema for commit shas, and one SessionUsage schema for what a counted session adds. The dashboard's 30-day window is the retention compaction keeps outcome lines for. A pull request lookup is dropped once its session is reported. Handle every attempt-outcome result explicitly, and share the pull request link markup in the dashboard. Refs Tencent#1387
…ass outcomes A session row leaves the dashboard list before its commits and pull request are counted, so the session detail and the dashboard guide point to the Overview panel. The workflow guide lists the pass outcomes a run record publishes; the knowledge guide says when a learning names its session; the engine notes record the attempt outcome file. Refs Tencent#1387
…s for it A resumed session whose earlier rollout compaction dropped was counted with only its remaining rollout. The outcome rows now take each session's totals from the usage report, dropped rollouts included. Refs Tencent#1387
git log reads --author as a basic regex, where the escaped + became a quantifier. Match the author as a fixed string. Refs Tencent#1387
…tree is gone A session that worked in a worktree removed before it was finished lost its commits. The report keeps the repository its checkout was in, and reads the commits from a live checkout of it when the session's own is gone. Refs Tencent#1387
…e chips in the session list A table-scroll wrapper was not positioned, so the absolutely positioned screen-reader text in its cells sized the page instead of the wrapper and the page scrolled sideways. The session list shows each outcome by kind and short sha; the full id stays in the chip title and in the details.
… find its pull request without origin HEAD A session that started on a checkout with no commit yet had no start head, so its commits were never read and it counted as nothing delivered. Its first start now bounds its commits by time when that start saw no head. The pull request lookup read the target branch from origin/HEAD, which only a clone sets, so a repository pushed to its origin was never looked up. It now resolves the default branch as the rest of teamai does.
… request merged before its session finished An earlier release recorded no Stop head and no learning for its sessions, so the first report after an upgrade counted each one already finished under `none`. With no reported-outcomes.json yet, those sessions are now flagged reported without being counted, and the file is written even when empty, so this happens once. A session's pull request was looked up once, when the session was finished, so one merged within those 6 hours was never found. Each report now looks it up while the session is still going, until it finds one.
A git killed at its 2-second cap was still waited for while a child of it held the pipe, as a wrapper's can, so the Stop handler ran past its limit and its event was never written. The cap now ends the wait, and the Stop is written without its head.
…ped on, not one found on an earlier branch A report now looks a session's pull request up while it is still going, so the answer it keeps may be for a branch the session later left. The kept answer now names its branch: a session that last stopped on another branch is asked about that one, and only the answer for that branch counts.
f3dbe8f to
c28caa2
Compare
All three earlier findings are resolved: first commits are detected, PR lookup no longer requires The description documents sufficient built real-CLI verification; no testing-record blocker. Deferring additional providers and agents to CI is acceptable. I did not run, build, or install PR code. |
… it stopped on A Stop on a detached HEAD records no branch, so the session keeps the branch before it, as during a rebase or bisect. The guides said "the branch the session last stopped on", which reads as none in that case.
|
Not changed: keeping the branch before a detached Stop is deliberate. The guides said "the branch the session last stopped on", which reads as no branch here. 24e26fc rewords them to "the last branch the session stopped on", and says a Stop on a detached HEAD keeps the branch before it. |
|
No new findings in the specified PR diff.
The description documents sufficient built real-CLI verification; no testing-record blocker. Deferring additional providers and agents to CI is acceptable. I did not run, build, or install PR code. |
SaulMoro
left a comment
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
Approving at 24e26fc after the rebase onto #1466. The conflicts were only in the dashboard's client.ts, locales.ts and styles.ts, and both sides are kept: the retry handler has both the outcomes and the scorers branches, switching workspace runs both resetScorers() and loadOutcomes(), and the duplicate SESSIONS translation was dropped in favour of #1466's. The other 19 commits are unchanged, and 24e26fc only rewords docs and comments.
We agree that keeping the branch before a detached Stop is right: a PR still counts only if origin's copy of that branch holds one of the session's commits, and clearing it would drop real PRs during a rebase.
At this head: npm ci, npm run build, npx tsc --noEmit, npm run lint exit 0; the unit files of this PR and #1466 (660 tests) pass, also with a symlinked TMPDIR; seven dashboard and report e2e files pass; a built-CLI run (hooks, contribute, two pulls with no double count, digest, /api/outcomes and /api/scorers) behaves as before the rebase.
…nd count it per run mode (Tencent#1471) Each session and workflow step now records what it delivered (a commit, a pull request, a learning, or nothing), and the usage report and digest count it per run mode. A session's pull request is the one of the branch it last stopped on, and sessions an earlier release finished are not counted. Refs Tencent#1387
Summary
The usage report counts sessions, prompts, tokens and interventions, but not what any of them delivered, so a session that shipped a fix and one that went nowhere look the same. Now a finished session, a workflow step pass and a run record what they delivered: commits, a pull request, learnings. The report and
teamai digestcount sessions, tokens and interventions per run mode and outcome. Ids, shas and links stay on the member's machine, where the dashboard shows them.Type of Change
Test Plan
npx tsc --noEmitpassesnpm run lintpassesnpx vitest runpasses on the files the last round changed (team-push-outcomes,dashboard-collector: 203 tests). The full suite passed before the rebase onto feat(score): lease a session, or a skill run, to the driver's own judge (#1386) #1470 (594 files, 17,558 tests). Locally (Node 24, macOS)workflow-reconciliationnow times out, on a cleanmaintoo, so CI's run covers this head.SESSIONStranslation (会话), since the dashboard has one translation per label. After the rebase, tsc and lint pass, and the dashboard, outcome, digest, hook, contribute and scorer tests pass (42 files, 776 tests). The built dashboard serves/api/outcomesand/api/scorers, and its page has both the outcomes and the scorers retry.Related e2e suites:
npm run test:e2e -- contribute-namespace dashboard-unified dashboard-workflow-runs dashboard-scoping report-scope-reuse workflow-records-reports-branch-1083 workflow-delivery-drivers workflow-session-1086 workflow-command.test(9 files, 55 tests pass).Real-CLI record on the built CLI (
npm run build). The sandbox has its ownHOME, a team repo with a bare origin (gitprovider) and an app repo with Claude hooks installed. Each session runs the installed hook commands (teamai hook-dispatch session-start | prompt-submit | stop --tool claude) with Claude hook payloads and a transcript, makes its commit during the session, and may runteamai contribute --session-id. Each session's events are then moved 7 hours back, so the member's nextteamai pullfinishes it.Upgrade from 0.28.0, in a fresh sandbox. The published 0.28.0 records three sessions through its hook commands (two commit, one also contributes a learning) and reports. Then the member reports on this build.
A Stop through
teamai hook-dispatch stop --tool claudewhile agitwrapper hangs on the Stop's head read and leaves a child holding the pipe:Not run here: a real pull request lookup. The
gitprovider opens none, so the dashboard's pull request row below comes from one seededsession_outcomeevent. The lookup itself (the branch the session last stopped on, after the checkout moved on; one of the session's commits; a PR found while the session went on and merged before it finished; the cache; no credentials) is covered byteam-push-outcomes.test.ts. Codex, CodeBuddy and OpenCode and thegitlabandgithubproviders are left to CI.Dashboard on the same sandbox. Desktop 1280:
zh-CN, dark, tablet 820 and mobile 430:
At 1280 the page no longer scrolls sideways. A table wrapper was not positioned, so the screen-reader text in its cells sized the page (
scrollWidth1392 before, 1280 after; 912 before and 415 after at 430). Wide tables scroll inside their panel.Related Issues
Part of #1387. Unattended sessions are counted like attended ones, 6 hours after they go quiet, at the member's next report. Counting them when
teamai score --drainends a CI job waits on #1385.Notes for Reviewers
session_outcomeevent per contribution and onecountedevent when the report finishes the session.readEventsskips them, so the live session fold is unchanged. A per-session flag inreported-outcomes.json, beside the existing baselines, keeps a session from being counted twice. A session's tokens and interventions are what the usage report itself counts for it, so the outcome rows agree with the top-level totals, including a rollout that event compaction dropped.launchedByis read for the session the outcome belongs to:contribute --session-idinside a workflow attempt still credits the member's session.--authorwith the git email as a fixed string, so a+tag matches), dated after the session's first start, ingitAtStart.head..<last Stop head>. A session that started with no commit yet (or whose git read failed at start) is bounded by its start time alone. A session whose checkout is gone is read in its repository, found from its project anchor. A session with no recorded Stop head gets no commits.getDefaultBranchresolves it (origin/HEAD, elseorigin/main, elseorigin/master), so a repository pushed to its origin rather than cloned is looked up too.pull), and once more when the session is finished. A session that later stops on another branch is asked about that branch, and only its answer counts. It uses the provider and credentialsdeliver: pruses, and keeps the answer inoutcome-prs.jsonuntil the session is reported. A failed lookup leavesprout and never fails the report.none. With noreported-outcomes.json, those sessions are flagged reported, and the file is written even when empty. The first report usually runs at the first session start after the upgrade, before any session on this release can be finished.