Repository navigation
feat(session): publish and score sessions per run mode, and drain them before an unattended job ends - #1478
Conversation
The PR description documents sufficient testing, including a built real-CLI verification record. No missing-e2e finding. I did not run, build or install PR code. |
b6b9993 to
363e663
Compare
|
Both earlier findings remain unresolved in the current diff:
The PR description documents sufficient testing, including built real-CLI verification; no missing-e2e finding. I found no additional confident issues and did not run, build or install PR code. |
Both earlier findings are resolved:
The PR description documents sufficient testing, including built real-CLI verification; no missing-e2e finding. I did not run, build or install PR code. |
The earlier single-secret title scenario, invalid-YAML wording, and PowerShell drain-command findings are resolved. The PR description documents sufficient testing, including built real-CLI verification; no missing-e2e finding. I did not run, build or install PR code. |
SaulMoro
left a comment
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
Reviewed at 5cc5cf4. Every #1385 acceptance item works, most of them checked with the built CLI. Two things need a change before merge: one is the bar for a team repo layout change, the other a regression in git error messages.
Blocking
-
[P1 blocking] The archive move needs a Breaking Changes line and the reverse mixed-release run. On one member HOME and one git-kind team, 0.28.0 pushed one session to the default branch, then this build pushed two to teamai-reports. This build lists all three, finds and resumes the old one, and deletes nothing on
main(git log --diff-filter=D mainis empty). Butsession liston 0.28.0 and on main prints1 session(s). A member who has not upgraded stops seeing every session an upgraded teammate archives, and by the code an olderworkflow resumedoes not restore step sessions archived there either. No data is lost, but every mixed-version team hits it, and the CHANGELOG lists the move only under### Changed. Please add a### 💥 Breaking Changesline, for example: "teamai session push,session migrateandworkflow publish --sessionsarchive on teamai-reports on every git-kind team. Members on 0.28.0 or earlier do not list, search or resume sessions archived there until they upgrade; sessions on the default branch stay readable to both." Then add the reverse direction (this build writes, 0.28.0 reads) to the body's record. -
[P2 non-blocking, but it fits this PR and should land with it] Git failures lose the ssh cause line.
GIT_PROBLEM_LINEinsrc/utils/branch-worktree.tskeeps onlyremote:,!,error:andfatal:lines, and ssh prints its cause without a prefix. With a stand-in ssh that denies the key,teamai score --pushprints onlyfatal: Could not read from remote repository.; main also printsgit@example.invalid: Permission denied (publickey).By the regex,Host key verification failed.andssh: Could not resolve hostname …are dropped too. This reaches every write to teamai-reports and teamai-learnings. Dropping progress lines instead of keeping a whitelist avoids it, for example:const GIT_PROGRESS_LINE = /^(?:To |Pushing to |hint:|Enumerating |Counting |Compressing |Writing |Total |remote: (?:Counting|Compressing|Resolving|Total)|$)/; const problems = message.split('\n').map((line) => line.trim().replace(/\s+/g, ' ')).filter((line) => line !== '' && !GIT_PROGRESS_LINE.test(line));
plus a
Permission denied (publickey)transcript inbranch-worktree-failure-reason.test.ts.
Non-blocking
- [P3 nit] Doctor says sessions are "offered for publishing in attended mode" under the default
ask, which does nothing until #1388. Printing the attended line only forauto, or saying "set to ask (not offered yet)", would avoid confusion. - [P3 nit]
score --drainprints each scoring-settings problem three times (judge, publish and usage paths each callscopeScoring). - [P3 nit] A batch
session pushwrites a second copy of a session already on the default branch without saying so. It is still listed once.
CI review findings: all three are fixed at this head and we checked each: the declared-secret title leak (reproduced at b6b9993, zero secrets at 5cc5cf4), the guide's claim that review --branch rejects invalid YAML (corrected), and the Windows PowerShell drain command (fixed by reading the code; Windows not run).
What we ran (built CLI, git-kind team on a local bare origin, base = main 3d253e4)
score --draintwice (idempotent), with the judge lock held (entries left open, exit 0), a refused push then a retry, and at the end of an unattended local workflow run.session push --session, scrubbed; bad values reported bydoctorand refused byreview --branch; member settings over the team's; attendedautoarchiving only the session idle over six hours.npx tsc --noEmitandnpm run lintexit 0; 35 changed unit files (995 tests) pass, also with a symlinked TMPDIR; 9 e2e files (20 tests) pass. On main, the PR's tests fail 26 cases, all new or changed by the PR.
Covered by unit tests only: namespace scoring files, the HTTP-team doctor note, a failed judgment, the judge-model-unavailable case.
ad4910f to
6c0604d
Compare
|
No new confident findings in the specified diff. All earlier findings are resolved:
The PR description documents sufficient testing, including built real-CLI verification. No missing-e2e finding; deferred provider/agent coverage is not blocking. I did not run, build, or install PR code. |
scoring.yaml and config.yaml scoring.sessions take off, ask or auto per run mode, or true/false for both. Each mode resolves on its own: member, then the most restrictive active namespace, then the root file, then the default (attended off, unattended auto). A bad value is reported with its key and allowed values and ignored. A captured session is judge work when the policy of the mode it was captured in is auto.
teamai.yaml and config.yaml take sessions.publish: { attended, unattended }
of off, ask or auto. Each mode resolves member, then team, then the default
(attended ask, unattended auto). A bad value is reported and ignored, and
never fails either file's load. An HTTP team publishes nothing.
doctor prints each sessions.publish value it ignores, and that an HTTP team publishes no sessions.
session push, session migrate and workflow publish --sessions now commit to teamai-reports on every git-kind team, as self mode already did. Readers look there first; on a git-kind team they also read the archive kept on the default branch before. session list and search show a session both hold once, from teamai-reports.
teamai session push --session <id> --source <platform> archives exactly that session, redacted and without prompting, on teamai-reports. pushSession is the programmatic entry it calls, returning archived, busy or failed.
session list and search read both archives through one helper now, so the SyncManager method has no caller. Its repo backfill is covered at the session list --all seam.
An unattended capture becomes judge work when teamai score --drain marks it drained, never after six idle hours. Attended sessions keep the six-hour rule.
score --drain marks the unattended captures of each reached data home finished, judges every judge entry a local tool here runs, settles a failed pass lease as failed so no drain judges it again, prints what failed and what is left for other drivers, and exits 1 when a judgment failed.
The drain publishes the labels it left pending, as score --push does, and exits 1 with the reason when they cannot reach teamai-reports. They stay pending, so the next drain publishes them.
With sessions.publish auto for the current run mode, the drain archives each finished session captured in that mode, scrubbed, on teamai-reports. A per-data-home record keeps an archived session from being pushed again; a push that fails records nothing, so the next drain retries it, and the drain prints one line for what stayed pending.
…drain With sessions.publish auto for the current run mode, the drain archives the session of each pass settled here in the ended runs of that mode, scrubbed, on teamai-reports, through the writer workflow publish --sessions uses. A pass whose tool teamai reads no sessions from is listed as skipped.
The drain reports the usage of each scope as pull does, honouring the usage report opt-out, so the counts of an ephemeral runner reach teamai-reports. A report that does not land exits 1.
Also covers that an attended drain archives only sessions idle six hours.
workflow start and resume with the local driver run the drain on the run's scope once an unattended run is done, after printing its outcome. The run's outcome keeps setting the exit code; what the drain could not do is printed.
With sessions.publish auto for attended sessions, the pass a session start begins archives the finished attended sessions of each scope before it judges, and a session start begins that pass for such a scope even when it scores no sessions.
Where sessions are published or scored in a run mode, doctor names each enabled tool teamai reads no sessions from, since its sessions are never captured and so are neither published nor judged.
…orts, as well as one archived on the default branch before The Scorers page read session names only from the team clone's archive, where a git-kind team no longer archives sessions. It now reads teamai-reports and the old archive, as session list does.
… cut inside it A title is cut to its length when its session is read, before the team's declared secrets are known. A declared secret that crossed the cut left its start in the archived title, _index.json and the file name, which the whole-value mask no longer matched. Scrubbing now masks that start as the cut would have had it known the secret.
…value It reports nothing for a teamai.yaml that does not parse.
…f them sets it PowerShell keeps $env:TEAMAI_MODE for the rest of the session, so after the command for the other mode, the bare command for this machine's mode drained the other mode's sessions again.
…g whole values shortens it A title that held a declared secret in full before the cut came out of the whole-value mask shorter than its cut length, so the start of a second copy at the cut was no longer checked and stayed in the archived title, _index.json and the file name.
…fails A failed push kept only remote:, !, error: and fatal: lines, so the line ssh prints without a prefix, such as Permission denied (publickey) or Host key verification failed, was dropped. The reason now drops git's progress lines and keeps every other one.
Members on 0.28.0 or earlier do not see sessions archived on teamai-reports until they upgrade.
…t to ask ask acts as off until sessions can be offered, so the attended note said sessions were offered for publishing when nothing was.
…ings Each lease a drain takes reads the scoring settings again, so a drain that judged printed each ignored value once a lease.
…anch also holds The push writes the session's current copy on teamai-reports, which session list and resume read first, and left the copy on the default branch unmentioned.
… in one The cause ssh gives ends a sentence of its own, so the drain and init printed two periods after it. A side-branch write reason drops its final period, and the drain drops it from every reason it lists.
01ac587 to
e0c0c53
Compare
|
All five items are fixed at e0c0c53, which is rebased on main after #1471.
The rebase onto #1471 needed one test change, 832762e. Its |
|
No new confident findings in the specified diff. All earlier findings are resolved:
The PR description documents sufficient testing, including built real-CLI verification. No missing-e2e finding; coverage explicitly deferred to CI is not blocking. I did not run, build, or install PR code. |
SaulMoro
left a comment
SaulMoro
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
Approving at e0c0c53. All five items from our last review are resolved, each checked with the built CLI.
| Item | Evidence |
|---|---|
| [P1] Breaking Changes line and the reverse mixed-release run | This build archives to teamai-reports; 0.28.0 and current main then list 1 session, find nothing with search and cannot resume the new one, while this build lists 2 and resumes it. Nothing on main is deleted. The Breaking line says exactly that. |
| [P2] ssh cause line | With a stand-in ssh that denies the key, score --push and score --drain print Permission denied (publickey). A push refused by a pre-receive hook still gives a short reason. |
[P3] doctor under attended ask |
The attended line is no longer printed for ask. |
| [P3] drain repeats settings warnings | Each warning prints once. |
| [P3] batch push re-archives a default-branch session | It now says so. |
The CI review's 20:14Z finding (part of a second copy of a declared secret left in a cut title) reproduces at the earlier head and is fixed by 2e4007a, with a unit test for that case.
One cosmetic nit: the joined ssh reason reads (publickey).; fatal: …; Please make sure you have the correct access rights; and the repository exists, with an extra .; and git's two-line hint split in the middle.
npm run build, npx tsc --noEmit and npm run lint exit 0; the 36 changed unit files (1,019 tests) pass, also with a symlinked TMPDIR; CI is green at this head.
@jeff-r2026 Breaking for the next tag: on every git-kind team, sessions are now archived on teamai-reports, so members on 0.28.0 or earlier cannot list, search or resume them (including step-session restore in workflow resume) until they upgrade; unattended runs now publish and score their sessions by default. See CHANGELOG.
Tencent#1478 added a member sessions setting to config.yaml, and Tencent#1473 made every config key either an install key or a member setting. Keep sessions per install, as scoring is, so the key ownership test passes again.
Summary
Sessions of a CI job or a cloud VM were never published or judged: they disappeared with the runner, so a team could not see how its agents did there. A team can now set, per run mode, whether sessions are published and scored (
off | ask | auto). Unattended runs default toautofor both, and attended runs stay as they were. Every unattended driver ends with one new step,teamai score --drain, which judges what this machine can, publishes, reports usage and says what is left. The session archive moves toteamai-reportsfor every git-kind team, and readers still find sessions archived on the default branch before.sessions.publish: { attended, unattended }teamai.yaml, memberconfig.yamlask(acts asoffuntil #1388),autosessions: { attended, unattended }(boolean still works)scorers/scoring.yaml, namespacescoring.yamloff,autoscoring.sessions: { attended, unattended }config.yamlEach mode resolves on its own: member, then the strictest active namespace, then root, then the default. A bad value is reported by
teamai doctorand ignored, andteamai review --branchfails ateamai.yamlthat has one.Also in this change:
teamai session push --session <id> --source <platform>pushes exactly that session, always scrubbed. The drain and attendedautopublishing use it.autopublishing runs in the background pass that judges finished sessions, once a session is six hours idle.teamai doctornames, once per run mode, the enabled tools teamai reads no sessions from.session pushsays when a session it pushes is also archived on the default branch:session listandresumeread the new copy onteamai-reportsfirst.teamai-reportsorteamai-learningsgives what went wrong on one line, the cause ssh prints included, without git's progress.teamai-reportsand from the archive on the default branch.env/secrets.yaml) as well, as judge evidence and run records already did. Titles are redacted before they are cut to length, and a title cut inside a declared secret is masked from where that secret starts, so a secret split by the cut no longer leaves its start in the title,_index.jsonor the file name.Dashboard
The built dashboard in headless Chrome, on a team with one attended session active and one finished, one unattended session waiting for a drain, one step session waiting to be archived and one pending label. No element in the panel overflows at any width, and the page is never wider than the viewport.
Whole overview at 1440: the panel sits under the three summary cards, before the trends and the sessions table.
Type of Change
Breaking: in CI, on a cloud VM or in a hosted job (
CIorTEAMAI_MODE=unattended), sessions are now archived onteamai-reportsand judged by every scorer withsessions: true. To keep the old behavior, setsessions: { publish: { unattended: off } }inteamai.yamlandsessions: { unattended: off }inscorers/scoring.yaml. The changelog and the scorers and sharing guides say so.Breaking:
session push,session migrateandworkflow publish --sessionsarchive onteamai-reportson every git-kind team. Members on 0.28.0 or earlier do not list, search or resume sessions archived there, and theirworkflow resumedoes not restore step sessions archived there, until they upgrade. Sessions on the default branch stay readable to both. The changelog lists it under Breaking Changes, and the dashboard guide says to upgrade every member.Test Plan
npx tsc --noEmit,npm run lint: clean.npx vitest runover the 181 test files this change touches or depends on: 4887 passed, 4 failed. The four are timeouts inworkflow-reconciliationandworkflow-settle-cost, which time out the same way onmainon this machine; none is an assertion on this change.npm run test:e2e -- dashboard-session-sharing score-sessions score-session session-publication workflow-publish-resume: 13 passed.claudethat logs every call, writes a transcript whose prompt carries aghp_token and judgesgood, and sessions captured through teamai's own hooks withCI=true. Scenarios 1 to 9 ran before the last three commits, which change two messages, the Chinese colon in the panel and an e2e test; those were checked with their unit and e2e tests and the screenshots above.session-title,score-drain,session-push,review-branchandagent-session-findpassing.dashboard-session-sharingunit tests (13 passed);npm run test:e2e -- dashboard-session-sharingpasses on the rebuilt head. Windows itself is not available here.bd3adf9b(feat(reporting): record what sessions and workflow steps delivered, and count it per run mode #1471, session outcomes):npx tsc --noEmitandnpm run lintclean; the tests of the files both sides touch pass (511), andnpm run test:e2e -- dashboard-session-sharing init-keeps-member-settings-1458 score-session score-sessions session-publication workflow-publish-resume dashboardpasses (26). Scenarios 12 and 13 ran on the built CLI before this rebase.40b0093b(fix(init): keep the member's own settings on init --force #1476):npm run test:e2e -- init-keeps-member-settings-1458passes with the member'ssessionssetting listed, and theinit,init-projectsandcommands-referenceunit tests pass.3d253e45(feat(dashboard): show scorer pass rates, what drives them, and whether a team change helped (#1182) #1466, the Scorers page):npx tsc --noEmitandnpm run lintclean, and the tests of the files both sides touch (dashboard-scorers,dashboard-ui,dashboard-session-sharing-view,dashboard-workspaces,scorers-results,contribute-reminder-record,score-drain,pull-scope-isolation,update,hook-handlers,score-sessions) pass.1. Unattended local run, no session settings
All three labels carry
runMode: unattended._index.jsontitles end in<REDACTED:gh_tok>.2. The same run with
unattended: offfor publishing and scoringThe run does not drain, and only two step calls reach
claude. An explicit drain still judges the step passes under their own scorers; the captured session stays unjudged and unarchived.3.
score --drainwith nothing to do, then a refused push, then againOrigin is unchanged after the refused drain, and
claudewas called to judge once in total: the pending label was published without judging again.4.
session push --sessionon a git-kind team5. Resuming a session the released CLI archived on the default branch
session listmerges both places: two sessions fromteamai-reportsand one frommain.6.
teamai doctorwith a bad value and tools teamai reads no sessions fromdoctorexits 1 here because the fixture listspiwithout installing it.7. A secret the team declares, with a name that does not look like one
The value is in no file, file name or commit on
teamai-reports.8.
score --drainwhile another judge pass on this machine runs9. A token that crosses the title length
The first prompt's token starts at character 48, so the title cut falls inside it. Before this change that title kept 19 characters of the token.
10. A declared secret that crosses the title cut
The prompt is a 48-character start followed by the value of
WAREHOUSE_DSN, declared inenv/secrets.yaml, so the 60-character title cut falls 12 characters into it.Without the fix, the new unit test fails because the scrubbed title keeps the start of the value.
11. A declared secret in full, then again across the title cut
The team declares
DSN. The prompt isUse mysql-prod-7Hq2xVb9Lk and compare with mysql-prod-7Hq2xVb9Lk, so the 60-character cut falls 17 characters into the second copy.Without the fix, the new unit test fails because the title ends in
mysql-prod-7Hq2xV.12. Mixed releases: this build archives, 0.28.0 reads
One member HOME and one git-kind team.
This is the Breaking Changes entry: 0.28.0 neither lists nor resumes what this build archives, and both read the default branch.
13. A batch push over the default branch, doctor under
ask, and a drain that cannot push over sshThe ignored value is printed once, where the drain printed it once for each lease it took. The ssh cause is kept, where the reason kept only the
fatal:line.Not run here, left to CI: the gitlab and github providers, Windows, and real Claude Code, Codex, CodeBuddy, OpenCode and WorkBuddy. The WorkBuddy and CodeBuddy IDE title cut is covered by unit tests only.
Related Issues
Closes #1385. Builds on #1384 (run mode) and #1386 (the driver's own judge, #1470).
askoffers nothing yet; #1388 adds it.Notes for Reviewers
src/score-drain.ts, called byscore --drainand by the local driver atdone.teamai score --sessionasks for a new judgment.TEAMAI_MODE=unattended teamai score --drain.session push -ywithout--scrubstill archives as-is behind its existing confirmation, as before. Every policy path andpush --sessionalways scrub.session push --scrub --repo-root <clone>masks secret-looking values but not the team's declared secrets: there is no teamai install to readenv/secrets.yamlfrom. The changelog says so.