feat(telemetry): track inline-completion and inline-chat acceptance separately#383
Open
pjdoland wants to merge 2 commits into
Open
feat(telemetry): track inline-completion and inline-chat acceptance separately#383pjdoland wants to merge 2 commits into
pjdoland wants to merge 2 commits into
Conversation
…eparately Add three telemetry event types so auto-complete and cell inline chat are recorded as distinct features, each with its own event identifier. inline-completion-accepted fires when a user takes an NBI inline completion. JupyterLab's inline completion provider interface has no accept callback, so the reliable signal is the inline completer widget's accept() method. Rather than subclass InlineCompleter (which would force us to reproduce JupyterLab's factory wiring: translator, toolbar buttons, keybinding hints), we wrap JupyterLab's own IInlineCompleterFactory, let it build the fully wired widget, and patch just the accept() method on the instance. The wrapper is installed on app.started: that resolves after every plugin has activated (so JupyterLab's own factory is already set and we win as the last writer) but before layout restoration creates the per-editor completer handlers, which capture the factory once at creation time and are never regenerated. Deferring to app.restored would be too late. inline-chat-accepted fires when generated code is applied, carrying a review|auto-insert mode discriminator. inline-chat-dismissed fires when the popover is torn down after code was shown but never applied, and only when the stream did not error (an interrupted stream surfaces a marker but no usable code, so tearing it down is a failure, not a user dismissal). Acceptance is tracked reliably; the ignore rate for auto-complete is derived in analysis as shown (inline-completion-response) minus accepted, which avoids a fragile dependency on JupyterLab-internal rejection signals. The completer-wrapping logic lives in src/inline-completer-telemetry.ts with a focused unit test, following the repo's extract-then-test pattern. The frontend (tokens.ts) and backend (api.py) TelemetryEventType enums are kept in parity.
The telemetry docs only described the chat-feedback (thumbs) hook, but NBI also emits inline-completion and inline-chat lifecycle events, now including inline-completion-accepted, inline-chat-accepted, and inline-chat-dismissed, all in-process. Generalize the admin guide's "Chat feedback event hook" section into "Telemetry events" with the full event-type list grouped by feature, and correct PRIVACY.md, which implied telemetry was limited to the gated feedback event. Clarify that the inline events fire on use and are not gated by enable_chat_feedback, while the privacy guarantee (nothing leaves the process without a custom listener) still holds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds telemetry so the two AI code-entry surfaces, auto-complete (inline completion) and cell inline chat, are tracked as distinct features with their own event identifiers. Three new
TelemetryEventTypevalues are introduced:inline-completion-accepted,inline-chat-accepted, andinline-chat-dismissed. This lets aTelemetryListenermeasure how often inline completions are accepted versus ignored, and how often inline-chat generations are applied versus dismissed, with auto-complete and inline chat cleanly separable.Solution
For inline completion, JupyterLab's provider interface has no accept callback, so the reliable accept signal is the inline completer widget's
accept()method. Rather than subclassInlineCompleter(which would force us to reproduce JupyterLab's factory wiring: translator, toolbar buttons, keybinding hints), we wrap JupyterLab's ownIInlineCompleterFactory, let it build the fully wired widget, and patch onlyaccept()on the instance. The wrapper installs onapp.started, which resolves after every plugin has activated (so JupyterLab's default factory is already set and ours wins as the last writer) but before layout restoration creates the per-editor completer handlers, which capture the factory once at creation and are never regenerated.IInlineCompleterFactoryis taken as an optional dependency, so NBI still loads if JupyterLab ever stops providing it.For inline chat,
inline-chat-acceptedfires when generated code is applied (with areviewvsauto-insertmode discriminator), andinline-chat-dismissedfires when the popover is torn down after code was shown but never applied. An interrupted stream surfaces a marker but no usable code, so its teardown is treated as a failure rather than a dismissal.Design note: we track acceptance reliably and derive the auto-complete ignore rate in analysis as shown (
inline-completion-response) minusinline-completion-accepted, rather than emitting an explicit ignore event. An explicit-ignore heuristic would depend on JupyterLab-internal rejection signals that are fragile across versions, whereas accept-plus-derive needs only events we emit ourselves.Testing
src/inline-completer-telemetry.tsis covered by a focused unit test (delegation to the wrapped factory, provider-gated emit, read-before-delegate ordering, lazy model-info read). Full suite green:jlpm tsc --noEmit,jlpm lint:check,jlpm jest(376 passing), andpytest(1235 passing). Verified live in a running JupyterLab against a real model:inline-completion-acceptedfires on Tab-accept with the completer toolbar UI preserved;inline-chat-acceptedfires for both auto-insert and review-accept;inline-chat-dismissedfires on review-cancel and is correctly suppressed when code was applied. The inline-chat flag logic lives in the plugin command closure and is exercised by the live checks rather than unit tests.Risks / follow-ups
Frontend (
tokens.ts) and backend (api.py) enums are kept in parity by hand; no consumer switches on the type, so adding values is backward compatible. Follow-ups noted during review, all pre-existing and out of scope here: inline-chat telemetry reportsNBIAPI.config.chatModeleven in Claude Code mode (every inline-chat event already does this, so it should be fixed holistically); thereview/auto-insertmode value could become a typed union alongside the existingeditorType/chatModestring discriminators.