Skip to content

fix(agent): record each Claude turn's own cost, not the session's running total - #2360

Merged
zhangfand merged 3 commits into
mainfrom
claude/project-thread-ys37d4
Nov 18, 2026
Merged

zhangfand merged 3 commits into
mainfrom
claude/project-thread-ys37d4

Conversation

@zhangfand

@zhangfand zhangfand commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Requested by zhangfan · project thread

Why

Rome records a cost per Claude turn in rome_agent_messages.cost_usd and sums those rows for a session's usage. The Agent SDK's total_cost_usd is not a turn's cost: it is the running total of the whole query() call, and Rome keeps one query open across a session's turns. So each turn recorded the spend of every turn before it too, and the sums overcounted. The chief of staff's test run showed it: the recorded turns summed to $7.24, while pricing their tokens gave about $0.07.

Invariants

  • A turn's recorded cost covers only that turn, so summing a session's turns gives the session's cost.
  • On a new thread, a turn's cost is what its result added to the SDK's running total.
  • On a resumed or forked thread, the first result's total includes turns already recorded, so that turn is priced from its tokens at Rome's rates, as when the SDK reports no cost. Later turns use the difference.
  • A result with no total, or the zero total the SDK puts on an empty or failed result, is priced from its tokens. The next total only sets where the following difference starts, so no turn is counted twice.
  • A turn the SDK starts on its own, such as a background task finishing, ends no Rome turn, so nothing records it. Its spend goes into the next Rome turn's difference. If the session closes before another Rome turn, that spend is not recorded.
  • When the total drops, as after /clear, the count starts over from that total. This is best effort: a /clear followed by a turn that ends above the old total undercounts that turn. Rome never sends /clear itself.

Scope

  • Anthropic provider only. Codex already reports a per-turn cost.
  • Rows recorded before this change keep their running totals. Nothing rewrites them.

Merge danger

Low. Usage and cost figures for Claude sessions go down to their true values from here on. Older rows still overcount.

Evidence

  • running-cost.test.ts covers a new thread, a resumed thread, a dropping total, a missing total and a zeroed placeholder.
  • A provider test resumes a thread whose first result reports a $42 running total and checks the turn is priced from its tokens instead. Another replays the SDK's empty task-notification result before the real one on a resumed thread. A third puts a costed background turn between two Rome turns and checks that the rows sum to the running total.
  • pnpm preflight passes.

🤖 Generated with Claude Code

…ning total

The Agent SDK's total_cost_usd is a running total for the query, so every
turn after the first recorded the spend of all turns before it, and sums
over turns grew quadratically. Record the difference from the previous
result. A resumed thread's first total includes turns already recorded,
so that turn is priced from its tokens instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zhangfand zhangfand self-assigned this Nov 18, 2026

@zoolsher zoolsher left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Design & Alternatives review: ✅ APPROVE

Problem: Rome keeps one Agent SDK query() open across a Claude session's turns. The SDK's total_cost_usd is the running total for that whole query, so every turn's rome_agent_messages.cost_usd row repeated the spend of the turns before it, and session totals came out far too high (the PR's test run: $7.24 recorded against about $0.07 of tokens).

Goals: each recorded row is the cost of that turn alone, so summing a session's rows gives its true cost. This should hold across new, resumed and branched threads, /clear, and results that carry no total. Non-goals: rewriting old rows, and the Codex provider (codex/app-server-provider.ts:1129 already prices each turn with calculateTurnCostUsd).

PR's approach: a small stateful RunningCost object for each openSession. On a new thread it starts at 0 and returns the difference between successive totals. On resume or branch it starts as unknown, so the first result returns undefined. buildAgentAccounting (provider-accounting.ts:313, reportedCostUsd ?? impliedCostUsd) then prices that result from its tokens. When the total drops, the count starts over from the new total. I checked the provider: every SDK result goes through the single runningCost.next(...) call at provider.ts:935, including results the SDK starts itself (sdkOwned). reopening-session.ts opens a new session with kind: "resume" each time, so it falls back to token pricing correctly.

Alternatives considered:

  1. Do nothing / fix it at the source. Rome can't change the SDK's meaning of total_cost_usd. Opening a new query() for every turn would undo the streaming-session design.
  2. Always price from tokens using the existing calculateImpliedCostUsd (provider-accounting.ts), ignoring total_cost_usd. This matches the Codex path and needs no state. But result.usage only covers the main loop, so it misses spend the SDK does count: subagents and side calls on other models, server-side tool fees, and pricing tiers missing from Rome's rate table. It also returns nothing for models the table doesn't know.
  3. Per-model differences from result.modelUsage[*].costUSD. More detail per model, but it is still a cumulative figure with the same resume problem, and it adds more code than the single scalar.
  4. Data-model change: store the raw running total plus a session marker, and compute differences when reading. This changes the schema (hard to undo) and every consumer of cost_usd, and old rows still can't be fixed without knowing where each query started.
Approach Added complexity Correctness/completeness Blast radius Reversibility Maintenance Effort
PR: running-total difference + token fallback on resume ~30 LOC, one class Per-turn cost including subagent and tool spend; first resumed turn is token-only Anthropic provider only Easy to undo, no schema or wire change Low Low
Do nothing None Overcounts more as a session goes on n/a n/a n/a None
Always price from tokens Less (delete the SDK cost) Misses spend outside the main loop and pricing tiers; nothing for unknown models Anthropic provider only Easy to undo Rate table must be kept up to date Very low
Per-model modelUsage difference More (a map of state) Similar to the PR, a bit more detail Anthropic provider only Easy to undo Medium Low–medium
Store raw total, compute differences when reading High Exact if markers are right; old rows still wrong DB schema, all cost readers Hard to undo (schema) High High

Recommendation: the PR's approach is the best fit. It fixes the overcount at the provider boundary where the SDK's meaning is known, and it reuses the existing reportedCostUsd ?? impliedCostUsd fallback instead of adding a new pricing path. It keeps the SDK's more complete cost for every turn except the first of a resumed thread, and it makes no schema or wire-format commitment. The one caveat is the rule that a drop means a restart, which is a heuristic (see the low-severity note).

Verdict: APPROVE — Taking the difference between successive SDK running totals, with the existing token-pricing fallback for the first result of a resumed thread, is the best fit. It is small and easy to reverse, and it keeps charges that token pricing alone would miss.

1 finding(s) posted as inline comments below.

Severity Category File Title
P3 design packages/core/src/agent/infrastructure/model/anthropic/running-cost.ts The reset heuristic undercounts when the first turn after a reset ends above the previous total

Automated review by RomeOS Code Review · commit 20c8458 · Full review history

@zoolsher zoolsher left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔁 This review has been superseded. See the latest review.

@zhangfand zhangfand left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔁 This review has been superseded. See the latest review.

A result with no total was priced from its tokens, but the next
difference still started from the total before it, so that turn was
counted twice. A zeroed placeholder result also set a resumed thread's
starting total to zero, so the next turn recorded the whole saved total.
Both now clear the starting total, and the next total only sets it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@zoolsher zoolsher left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔁 This review has been superseded. See the latest review.

@zhangfand zhangfand left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔁 This review has been superseded. See the latest review.

An SDK-owned result ends no Rome turn, so its accounting is never
recorded. Moving the cost tracker past it lost that spend. The tracker
now skips such results, so the next Rome turn's difference includes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@zoolsher zoolsher left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review: ✅ APPROVE

Goal: The Agent SDK's total_cost_usd is a running total for the whole query() call. This PR turns it into the cost of each Claude turn, so summing a session's rome_agent_messages.cost_usd rows gives the session's real cost.

Changes since the last review (a356eab):

  • An SDK-owned result no longer moves the tracker: line 938 has sdkOwned ? undefined : runningCost.next(...). Background spend now lands in the next Rome turn's difference. The new provider test runs totals $0.50 (Rome), $0.75 (background), $1.00 (Rome) and checks that each Rome turn records $0.50. This fixes the P2 that both earlier reviews raised.
  • A zeroed placeholder can't move the starting point either, because placeholders come in as SDK-owned task-notification results. The total === 0 && last !== 0 check still covers zeroed results that Rome owns.

What I checked beyond the diff:

  • buildAgentAccounting uses reportedCostUsd ?? impliedCostUsd, so a real difference of 0 is kept and not replaced by the token price. undefined falls back to the token price, as intended.
  • readPositiveNumber accepts 0, so the placeholder branch can actually be reached.
  • total_cost_usd has no other readers in packages/, so there's no second path that still sums running totals.
  • RunningCost is created once for each openSession. params.thread.kind === "new" is the only case where the SDK starts its total at zero. resume and branch (fork) both start with no known starting point.
  • For SDK-owned results, resultData.costUsd (logging only) now shows the token price instead of the running total. That's harmless.
  • I traced every case in running-cost.test.ts by hand and they match the code. I couldn't run the test suites because the clone has no dependencies installed.

Design: A small stateful object for each session, falling back to token pricing when the starting point is unknown, is the simplest fix that works. The alternative, saving SDK-owned results as their own rows, would mean adding a new event or persistence path just for background turns. The author chose to fold that spend into the next Rome turn, which keeps the change small. The trade-off is the trailing-spend case in the single P3 finding.

The earlier /clear notes are now explained in the code comment, so I'm not raising them again.

Verdict: APPROVE — I found no P0–P2 problems. The fix records each Claude turn's own cost, and both P2 findings from earlier reviews are fixed. The one note left is a low-priority (P3) edge case.

1 finding(s) posted as inline comments below.

Severity Category File Title
P3 correctness packages/core/src/agent/infrastructure/model/anthropic/provider.ts Background spend is still lost when no later Rome turn can take a difference

Automated review by RomeOS Code Review · commit a356eab · Full review history

// An SDK-owned result ends no Rome turn, so nothing records its
// accounting. Leaving the tracker where it was puts its spend in
// the next Rome turn's difference.
sdkOwned ? undefined : runningCost.next(readPositiveNumber(message.total_cost_usd)),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] correctness — Background spend is still lost when no later Rome turn can take a difference

An SDK-owned turn's spend only gets recorded when a later Rome result takes a difference from a known starting total. In three cases it is lost:

  • The session closes after a background turn and before the next Rome turn.
  • On a resumed or forked thread, the background turn comes before the first Rome result. That result is priced from its own usage tokens only.
  • The next Rome result has no total.

This isn't a regression, since trailing background spend was never recorded before either, and the amounts are small. If you want it to be exact, you could add the token price of skipped SDK-owned results to the next Rome turn whenever that turn falls back to token pricing. Otherwise, mention the gap in the comment.

@zhangfand zhangfand left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review: ✅ APPROVE

Reviewed all four changed files and traced session lifecycle, pricing fallback, and accounting consumers. The implementation is appropriately contained within the Anthropic adapter.

All six tracker scenarios passed direct Node assertions. The provider tests could not run because dependencies, including rstest, are not installed.

Verdict: APPROVE — The query-scoped cost tracker correctly implements the intended per-turn accounting without actionable regressions.

No issues found. Code looks good!


Automated review by RomeOS Code Review · commit a356eab

@zhangfand
zhangfand merged commit c8bfc0e into main Nov 18, 2026
15 of 21 checks passed
@zhangfand
zhangfand deleted the claude/project-thread-ys37d4 branch November 18, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants