Skip to content

feat(telemetry): track inline-completion and inline-chat acceptance separately#383

Open
pjdoland wants to merge 2 commits into
plmbr:mainfrom
pjdoland:feat/inline-completion-acceptance-telemetry
Open

feat(telemetry): track inline-completion and inline-chat acceptance separately#383
pjdoland wants to merge 2 commits into
plmbr:mainfrom
pjdoland:feat/inline-completion-acceptance-telemetry

Conversation

@pjdoland

Copy link
Copy Markdown
Collaborator

Summary

Adds telemetry so the two AI code-entry surfaces, auto-complete (inline completion) and cell inline chat, are tracked as distinct features with their own event identifiers. Three new TelemetryEventType values are introduced: inline-completion-accepted, inline-chat-accepted, and inline-chat-dismissed. This lets a TelemetryListener measure how often inline completions are accepted versus ignored, and how often inline-chat generations are applied versus dismissed, with auto-complete and inline chat cleanly separable.

Solution

For inline completion, JupyterLab's provider interface has no accept callback, so the reliable accept signal is the inline completer widget's accept() method. Rather than subclass InlineCompleter (which would force us to reproduce JupyterLab's factory wiring: translator, toolbar buttons, keybinding hints), we wrap JupyterLab's own IInlineCompleterFactory, let it build the fully wired widget, and patch only accept() on the instance. The wrapper installs on app.started, which resolves after every plugin has activated (so JupyterLab's default factory is already set and ours wins as the last writer) but before layout restoration creates the per-editor completer handlers, which capture the factory once at creation and are never regenerated. IInlineCompleterFactory is taken as an optional dependency, so NBI still loads if JupyterLab ever stops providing it.
For inline chat, inline-chat-accepted fires when generated code is applied (with a review vs auto-insert mode discriminator), and inline-chat-dismissed fires when the popover is torn down after code was shown but never applied. An interrupted stream surfaces a marker but no usable code, so its teardown is treated as a failure rather than a dismissal.
Design note: we track acceptance reliably and derive the auto-complete ignore rate in analysis as shown (inline-completion-response) minus inline-completion-accepted, rather than emitting an explicit ignore event. An explicit-ignore heuristic would depend on JupyterLab-internal rejection signals that are fragile across versions, whereas accept-plus-derive needs only events we emit ourselves.

Testing

src/inline-completer-telemetry.ts is covered by a focused unit test (delegation to the wrapped factory, provider-gated emit, read-before-delegate ordering, lazy model-info read). Full suite green: jlpm tsc --noEmit, jlpm lint:check, jlpm jest (376 passing), and pytest (1235 passing). Verified live in a running JupyterLab against a real model: inline-completion-accepted fires on Tab-accept with the completer toolbar UI preserved; inline-chat-accepted fires for both auto-insert and review-accept; inline-chat-dismissed fires on review-cancel and is correctly suppressed when code was applied. The inline-chat flag logic lives in the plugin command closure and is exercised by the live checks rather than unit tests.

Risks / follow-ups

Frontend (tokens.ts) and backend (api.py) enums are kept in parity by hand; no consumer switches on the type, so adding values is backward compatible. Follow-ups noted during review, all pre-existing and out of scope here: inline-chat telemetry reports NBIAPI.config.chatModel even in Claude Code mode (every inline-chat event already does this, so it should be fixed holistically); the review/auto-insert mode value could become a typed union alongside the existing editorType/chatMode string discriminators.

…eparately

Add three telemetry event types so auto-complete and cell inline chat are
recorded as distinct features, each with its own event identifier.

inline-completion-accepted fires when a user takes an NBI inline completion.
JupyterLab's inline completion provider interface has no accept callback, so
the reliable signal is the inline completer widget's accept() method. Rather
than subclass InlineCompleter (which would force us to reproduce JupyterLab's
factory wiring: translator, toolbar buttons, keybinding hints), we wrap
JupyterLab's own IInlineCompleterFactory, let it build the fully wired widget,
and patch just the accept() method on the instance. The wrapper is installed on
app.started: that resolves after every plugin has activated (so JupyterLab's
own factory is already set and we win as the last writer) but before layout
restoration creates the per-editor completer handlers, which capture the
factory once at creation time and are never regenerated. Deferring to
app.restored would be too late.

inline-chat-accepted fires when generated code is applied, carrying a
review|auto-insert mode discriminator. inline-chat-dismissed fires when the
popover is torn down after code was shown but never applied, and only when the
stream did not error (an interrupted stream surfaces a marker but no usable
code, so tearing it down is a failure, not a user dismissal).

Acceptance is tracked reliably; the ignore rate for auto-complete is derived in
analysis as shown (inline-completion-response) minus accepted, which avoids a
fragile dependency on JupyterLab-internal rejection signals.

The completer-wrapping logic lives in src/inline-completer-telemetry.ts with a
focused unit test, following the repo's extract-then-test pattern. The frontend
(tokens.ts) and backend (api.py) TelemetryEventType enums are kept in parity.
@pjdoland pjdoland added the enhancement New feature or request label Jun 26, 2026
The telemetry docs only described the chat-feedback (thumbs) hook, but NBI also
emits inline-completion and inline-chat lifecycle events, now including
inline-completion-accepted, inline-chat-accepted, and inline-chat-dismissed, all
in-process. Generalize the admin guide's "Chat feedback event hook" section into
"Telemetry events" with the full event-type list grouped by feature, and correct
PRIVACY.md, which implied telemetry was limited to the gated feedback event.
Clarify that the inline events fire on use and are not gated by
enable_chat_feedback, while the privacy guarantee (nothing leaves the process
without a custom listener) still holds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant