Skip to content

feat(server): count which product features people use - #15921

Open
juliusmarminge wants to merge 1 commit into
mainfrom
t3code/feature-usage-analytics
Open

juliusmarminge wants to merge 1 commit into
mainfrom
t3code/feature-usage-analytics

Conversation

@juliusmarminge

@juliusmarminge juliusmarminge commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Today analytics only counts connections, threads and turns. We can't tell which product features are used, such as checkpoint restore, fork, provider switch, the terminal, the preview browser, worktrees, git and PR actions, or scheduled tasks.

This adds one anonymous PostHog event, feature.used { feature, variant?, surface? }. It's recorded from the RPC authorization middleware, which every WebSocket request already passes through, after the request succeeds (for a stream RPC, when the stream completes). No handler changes. rpcScopeAuthorizationLayer gains an optional onAuthorized callback, and ws.ts passes one that maps the request through FeatureUsage.ts.

FeatureUsage.ts is an allowlist:

  • User commands via dispatchCommand: checkpoint.rollback (with_files / conversation_only), thread.fork, thread.merge_back, provider.switch, archive, snooze, pin, settle, PR link and watch, runtime and interaction mode changes (mode as variant), interrupt, queue steer and edit. message.dispatch is left out because client.turn.requested already counts it. Housekeeping commands such as visits and reorders are left out too.
  • RPCs: thread.launch (workspace strategy as variant), terminal, preview and device open, editor.open (editor id), worktree create, git.stacked_action (action), pull_request.action (action), PR review and comment, scheduled task save and run-now, agent session import, project clone, repository publish, keybinding customize.

Anonymity: variants come only from closed contract enums. No paths, ids, titles, prompts or free text, and the only client dimension is surface. Hot paths such as terminal.write don't map to a feature and return right away. T3CODE_TELEMETRY_ENABLED=false still disables everything.

Tests: FeatureUsage.test.ts checks the mapping, including that paths, prompts and ids never appear. RpcAuthorization.test.ts checks that only authorized, successful requests are reported. The existing ws tests pass.

Companion to #15913, which covers the agent side (MCP tools and delegation). The two branches are independent.

Done by Claude Opus 5.5 in Claude Code (via T3 Code).

🤖 Generated with Claude Code


Devin Review

Records an anonymous feature.used event for allowlisted feature RPCs
and user commands (checkpoint restore, fork, provider switch, terminal,
preview, worktrees, git actions, PR actions, scheduled tasks, ...),
with a closed-enum variant and the client surface. Recorded from the
RPC authorization middleware so no handler changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. label Oct 5, 2026
@juliusmarminge juliusmarminge added the macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews label Oct 5, 2026
@github-actions github-actions Bot added the size:L 100-499 changed lines (additions + deletions). label Oct 5, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds production feature-usage telemetry to authorized WebSocket requests and modifies shared authentication middleware. Because files under the auth directory are changed, the mandatory sensitive-package review rule requires human review.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

⚠️ The thread fixture changed, so impact percentages are not directly comparable to the main baseline.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire — 5.0 KiB — 6.8 KiB ✅
Codex Thread snapshot wire — 3.8 KiB — 4.9 KiB ✅
Codex Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Codex Live turn WebSocket decoded — 20.9 KiB — 29.3 KiB ✅
Codex Live turn messages — 2 — 8 ✅
Claude Total thread wire — 5.0 KiB — 6.8 KiB ✅
Claude Thread snapshot wire — 3.8 KiB — 4.9 KiB ✅
Claude Live turn WebSocket wire — 1.2 KiB — 2.0 KiB ✅
Claude Live turn WebSocket decoded — 21.2 KiB — 29.3 KiB ✅
Claude Live turn messages — 1 — 8 ✅

Baseline: 2efb817 · PR result: b171752 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@coderabbitai

coderabbitai Bot commented Oct 5, 2026

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

🧰 Additional context used
📚 Code guidelines (2)
docs/internals/effect-services.md — auto-discovered
AGENTS.md — auto-discovered

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: pingdotgg/t3code/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Team
  • Run ID: 054b8ec9-b74a-4fe9-9c7b-c1d3e4a3e33c
📥 Commits

Reviewing files that changed from the base of the PR and between 06e6274 and b171752.

📒 Files selected for processing (6)
  • apps/server/src/auth/RpcAuthorization.test.ts
  • apps/server/src/auth/RpcAuthorization.ts
  • apps/server/src/telemetry/FeatureUsage.test.ts
  • apps/server/src/telemetry/FeatureUsage.ts
  • apps/server/src/ws.ts
  • docs/internals/product-analytics.md

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.


📝 Walkthrough

Walkthrough

The server classifies selected WebSocket RPC requests as feature usage. After an authorized request succeeds, the server records classified usage with the connecting client’s surface when available. Tests cover classification and callback behavior.

Changes

Feature Usage Analytics

Layer / File(s) Summary
Feature classification
apps/server/src/telemetry/FeatureUsage.ts, apps/server/src/telemetry/FeatureUsage.test.ts
Adds mappings for selected commands and RPCs. Variants use selected string values, and untracked requests return undefined. Tests cover tracked features, variants, and excluded payloads.
Post-success authorization callback
apps/server/src/auth/RpcAuthorization.ts, apps/server/src/auth/RpcAuthorization.test.ts
Adds an optional callback that receives the method and payload after an authorized request succeeds. Tests confirm that a request rejected for insufficient scope is not recorded.
WebSocket analytics recording
apps/server/src/ws.ts, docs/internals/product-analytics.md
Connects the callback to feature classification and feature.used recording. Documents when usage is recorded and which data is included.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant WebSocketRpcRoute
  participant rpcScopeAuthorizationLayer
  participant featureUsage
  participant FeatureUsedRecorder
  WebSocketRpcRoute->>rpcScopeAuthorizationLayer: Pass RPC method and payload
  rpcScopeAuthorizationLayer->>rpcScopeAuthorizationLayer: Authorize and run request
  rpcScopeAuthorizationLayer->>featureUsage: Classify successful request
  featureUsage-->>rpcScopeAuthorizationLayer: Return feature data or undefined
  rpcScopeAuthorizationLayer->>FeatureUsedRecorder: Record feature.used when classified
Loading

Suggested reviewers: t3dotgg

Merge Risk: ⚪ Minimal · up to b1717

No actionable issue is established for this change. Stream completion timing remains unverified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description explains the problem, implementation, telemetry exclusions, and tests. However, it does not provide the required scope and approval information for this feature change. Mentioning comp… Add a Scope and approval section with a link to the triaged issue or discussion and the maintainer’s explicit approval of this direction and scope. If this change qualifies for an exemption, explain why it is a very small, focused fix of an…
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: tracking product feature usage on the server.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 5 files. (1 skipped: 1 …
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description explains the problem, implementation, telemetry exclusions, and tests. However, it does not provide the required scope and approval information for this feature change. Mentioning companion PR #15913 does not establish maintainer approval for this PR.

Resolution

Add a Scope and approval section with a link to the triaged issue or discussion and the maintainer’s explicit approval of this direction and scope. If this change qualifies for an exemption, explain why it is a very small, focused fix of an obvious bug; otherwise, provide the required approval evidence.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews size:L 100-499 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant