Repository navigation
fix(usage): stop charging L3 reuse cache-hit tokens as paid cost (Issue #548) - #555
Merged
Merged
Conversation
#548) Signed-off-by: Ryo Kusnadi <ryokusnadi@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Problem:
semantix usageand the dashboard show a negative paid cost and savings above 100% when the log has gateway L3 reuse turns. The issue's example (1000 in, 800 cache hit, L3 reuse) givespaid=-0.00016and a 159% savings rate. Mixed logs also under-report what the upstream turns really cost.How it happens: when the gateway answers from a stored result, it writes a usage event with
l3_reuse: trueandcache_hit_tokensset to the size of the reused answer (gateway/pipeline.go, therecordUsagecall in the L3 branch). That is a local reply, not a provider prefix-cache hit. Anysemantix usagerun on a log with such a turn hits it.Cause: in
kernel/usage/usage.goSummarize, billed input isTokensIn - CacheHitTokens - l3In, and the hit bucket is chargedCacheHitTokens * hit price. For an L3 turn the hit tokens are already insidel3In, so they are subtracted twice from the miss bucket and still charged at the hit price. Each L3 turn adds a negative-(hits)*(miss-hit)/1e6.Fix: sum the cache-hit tokens of L3 turns (
l3Hit) and charge onlyCacheHitTokens - l3Hitat the hit price and subtract only that from billed input. L3 turns now leave every billed bucket. I did not clamp the total at zero, as the issue asks, so mixed logs stay correct. TheCacheHitTokensfigure in the summary is unchanged.Scope
kernel/usageonly (Summarize).InjectROIalready counts L3 savings froml3In/l3Outand is not affected. Per-provider stats only countExactevents and L3 events are not exact, so they are not affected either.Validation
Before the fix:
TestSummarizeL3OnlyCostsZerofails withpaid = -0.00015999999999999999, want 0, andTestSummarizeL3MixedKeepsUpstreamCostfails withpaid = 4.400000000000001e-05, want 0.000204.After: both pass.
Ran:
go vet ./kernel/usage/,go test ./kernel/usage/ -race,go test ./cmd/semantix/,gofmt -l kernel/usage,git diff --check.Not run:
go vet ./...andgo test ./... -racefor the whole repo.go vet ./...(run on./kernel/usage/only)go test ./... -race(run on./kernel/usage/only, plus./cmd/semantix/without-race)git diff --checkAdditional checks are documented below, or are not applicable.
Compatibility and data impact
None for the log format.
CostPaidUSD,SavingsUSDandSavingsRatechange for logs that contain L3 reuse turns, to the correct values.Security and privacy
None.
Evidence
Issue example (one L3 turn, 1000 in, 800 hit): paid
-0.00016and rate1.59before; paid0and rate1after.Related issues
Closes #548
Checklist
docs/events.md.