From 7610f9ee5b6d6711b185264aa70ccced531af054 Mon Sep 17 00:00:00 2001 From: "posthog[bot]" <206114724+posthog[bot]@users.noreply.github.com> Date: Sun, 6 Sep 2026 15:56:51 +0000 Subject: [PATCH 1/7] fix(replay-vision): trim repeated fields from scanner observation rows The scanner-scoped observations list re-embedded the scanner's prompt and recordings query on every row, and restated each row's reasoning a second time as reasoning_segments. On the default page this overflowed an agent's token budget, so the tool could not be called without first guessing a small limit. Declare the three paths under `response.exclude` on the MCP tool, and default its `limit` to 20. The trim sits in the MCP layer because the app's own observations table, calibration tab, replay dock, and clipboard export all read those fields from list rows; the REST API keeps serving them. Point the tool description and agent note at vision-scanners-observations-get for one observation's full prompt and citation timestamps. Generated-By: PostHog Desktop Task-Id: cfd06794-5dc1-4cdb-b3d1-f66d05273694 --- products/replay_vision/mcp/tools.yaml | 24 ++++-- .../schema/generated-tool-definitions.json | 2 +- services/mcp/schema/tool-definitions-all.json | 2 +- .../mcp/src/tools/generated/replay_vision.ts | 35 +++++++-- .../vision-scanners-observations-list.json | 3 +- ...scanner-observations-list-trimming.test.ts | 73 +++++++++++++++++++ 6 files changed, 124 insertions(+), 15 deletions(-) create mode 100644 services/mcp/tests/unit/vision-scanner-observations-list-trimming.test.ts diff --git a/products/replay_vision/mcp/tools.yaml b/products/replay_vision/mcp/tools.yaml index e7279ad57ab1..81a435fdd054 100644 --- a/products/replay_vision/mcp/tools.yaml +++ b/products/replay_vision/mcp/tools.yaml @@ -451,12 +451,26 @@ tools: `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. + Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and + `scanner_snapshot.query` are the same on every row, and `model_output.reasoning_segments` restates the + flat `reasoning`, so none of the three are on a row. Call `vision-scanners-observations-get` for one + observation's full prompt and citation segments, or `vision-scanners-get` for the scanner's current + prompt. + param_overrides: + limit: + description: Number of observations to return per page (default 20). + default: 20 + response: + exclude: + - scanner_snapshot.scanner_config.prompt + - scanner_snapshot.query + - scanner_result.model_output.reasoning_segments agent_note: > - Each observation's `_posthogUrl` opens the recording it analysed. - `scanner_result.model_output.reasoning_segments` interleaves prose with `chip` segments, and a chip's - `timestamp_ms` is the recording-relative offset of the moment being cited — append `?t=` - (`timestamp_ms` / 1000, rounded down) to that URL to seek straight to it. When you report a finding to - someone, deep-link the one or two moments it turns on rather than only describing them. + Each observation's `_posthogUrl` opens the recording it analysed. This list carries no citation + timestamps. When you report a finding to someone, deep-link the one or two moments it turns on rather + than only describing them: read that observation with `vision-scanners-observations-get`, take a `chip` + segment's `timestamp_ms` from `scanner_result.model_output.reasoning_segments`, and append + `?t=` (`timestamp_ms` / 1000, rounded down) to the recording URL. ui_app: vision-observation-list url_prefix: /replay vision-scanners-observations-retry-create: diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json index 05d3dbfca5b7..315a70e21e32 100644 --- a/services/mcp/schema/generated-tool-definitions.json +++ b/services/mcp/schema/generated-tool-definitions.json @@ -11673,7 +11673,7 @@ } }, "vision-scanners-observations-list": { - "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`.", + "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` are the same on every row, and `model_output.reasoning_segments` restates the flat `reasoning`, so none of the three are on a row. Call `vision-scanners-observations-get` for one observation's full prompt and citation segments, or `vision-scanners-get` for the scanner's current prompt.", "category": "Replay vision", "feature": "replay_vision", "summary": "List scanner observations", diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json index 112be5747f6f..734e75aef699 100644 --- a/services/mcp/schema/tool-definitions-all.json +++ b/services/mcp/schema/tool-definitions-all.json @@ -12301,7 +12301,7 @@ } }, "vision-scanners-observations-list": { - "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`.", + "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` are the same on every row, and `model_output.reasoning_segments` restates the flat `reasoning`, so none of the three are on a row. Call `vision-scanners-observations-get` for one observation's full prompt and citation segments, or `vision-scanners-get` for the scanner's current prompt.", "category": "Replay vision", "feature": "replay_vision", "summary": "List scanner observations", diff --git a/services/mcp/src/tools/generated/replay_vision.ts b/services/mcp/src/tools/generated/replay_vision.ts index 3d162ec65094..ee192f21ef6c 100644 --- a/services/mcp/src/tools/generated/replay_vision.ts +++ b/services/mcp/src/tools/generated/replay_vision.ts @@ -4,7 +4,13 @@ import { z } from 'zod' import type { Schemas } from '@/api/generated' import * as orvalSchemas from '@/generated/replay_vision/api' import { withUiApp } from '@/resources/ui-apps' -import { withPostHogUrl, withAgentNote, type WithPostHogUrl, type WithAgentNote } from '@/tools/tool-utils' +import { + withPostHogUrl, + withAgentNote, + omitResponseFields, + type WithPostHogUrl, + type WithAgentNote, +} from '@/tools/tool-utils' import type { Context, ToolBase, ZodObjectAny } from '@/tools/types' const VisionObservationsLabelCreateSchema = () => { @@ -515,9 +521,14 @@ const visionScannersObservationsGet = (): ToolBase< const VisionScannersObservationsListSchema = () => { const VisionScannersObservationsListParams = orvalSchemas.VisionScannersObservationsListParams() const VisionScannersObservationsListQueryParams = orvalSchemas.VisionScannersObservationsListQueryParams() - return VisionScannersObservationsListParams.omit({ project_id: true }).extend( - VisionScannersObservationsListQueryParams.shape - ) + return VisionScannersObservationsListParams.omit({ project_id: true }) + .extend(VisionScannersObservationsListQueryParams.shape) + .extend({ + limit: VisionScannersObservationsListQueryParams.shape['limit'] + .default(20) + .optional() + .describe('Number of observations to return per page (default 20).'), + }) } const visionScannersObservationsList = (): ToolBase< @@ -550,20 +561,30 @@ const visionScannersObservationsList = (): ToolBase< verdict: params.verdict, }, }) + const filtered = { + ...result, + results: (result.results ?? []).map((item: any) => + omitResponseFields(item, [ + 'scanner_snapshot.scanner_config.prompt', + 'scanner_snapshot.query', + 'scanner_result.model_output.reasoning_segments', + ]) + ), + } as typeof result return withAgentNote( await withPostHogUrl( context, { - ...result, + ...filtered, results: await Promise.all( - (result.results ?? []).map((item) => + (filtered.results ?? []).map((item) => withPostHogUrl(context, item, `/replay/${item.session_id}`) ) ), }, '/replay' ), - "Each observation's `_posthogUrl` opens the recording it analysed. `scanner_result.model_output.reasoning_segments` interleaves prose with `chip` segments, and a chip's `timestamp_ms` is the recording-relative offset of the moment being cited — append `?t=` (`timestamp_ms` / 1000, rounded down) to that URL to seek straight to it. When you report a finding to someone, deep-link the one or two moments it turns on rather than only describing them.\n" + "Each observation's `_posthogUrl` opens the recording it analysed. This list carries no citation timestamps. When you report a finding to someone, deep-link the one or two moments it turns on rather than only describing them: read that observation with `vision-scanners-observations-get`, take a `chip` segment's `timestamp_ms` from `scanner_result.model_output.reasoning_segments`, and append `?t=` (`timestamp_ms` / 1000, rounded down) to the recording URL.\n" ) }, }) diff --git a/services/mcp/tests/unit/__snapshots__/tool-schemas/vision-scanners-observations-list.json b/services/mcp/tests/unit/__snapshots__/tool-schemas/vision-scanners-observations-list.json index 78c86de5b216..51b399219eef 100644 --- a/services/mcp/tests/unit/__snapshots__/tool-schemas/vision-scanners-observations-list.json +++ b/services/mcp/tests/unit/__snapshots__/tool-schemas/vision-scanners-observations-list.json @@ -18,7 +18,8 @@ "type": "boolean" }, "limit": { - "description": "Number of results to return per page.", + "default": 20, + "description": "Number of observations to return per page (default 20).", "type": "number" }, "max_score": { diff --git a/services/mcp/tests/unit/vision-scanner-observations-list-trimming.test.ts b/services/mcp/tests/unit/vision-scanner-observations-list-trimming.test.ts new file mode 100644 index 000000000000..c0108af69680 --- /dev/null +++ b/services/mcp/tests/unit/vision-scanner-observations-list-trimming.test.ts @@ -0,0 +1,73 @@ +import { describe, expect, it } from 'vitest' + +import { GENERATED_TOOLS } from '@/tools/generated/replay_vision' +import type { Context, ToolBase, ZodObjectAny } from '@/tools/types' + +// One observation row carrying every field the tool's `response.exclude` drops, alongside the +// siblings it must keep, so a broken path (silent no-op -> full payload back) is visible. +const FULL_ROW = { + id: 'obs-1', + session_id: 'sess-1', + status: 'succeeded', + scanner_snapshot: { + name: 'Checkout drop-offs', + scanner_type: 'monitor', + scanner_version: 3, + // Identical on every row of a scanner's page, and the largest field on it. + scanner_config: { prompt: 'Did the shopper abandon the cart?', allow_inconclusive: true }, + query: { kind: 'RecordingsQuery', events: [{ id: '$pageview' }] }, + }, + scanner_result: { + model_output: { + scanner_type: 'monitor', + verdict: 'yes', + reasoning: 'The shopper opened the cart and left without paying.', + // Restates `reasoning` interleaved with timestamp chips. + reasoning_segments: [ + { kind: 'text', value: 'The shopper opened the cart and left without paying.' }, + { kind: 'chip', timestamp_ms: 42000 }, + ], + }, + signals_count: 0, + }, +} + +function mockContext(): Context { + return { + stateManager: { getProjectId: async () => 1 }, + api: { + getProjectBaseUrl: () => 'https://us.posthog.com/project/1', + request: async () => ({ count: 1, next: null, previous: null, results: [structuredClone(FULL_ROW)] }), + }, + } as unknown as Context +} + +describe('vision-scanners-observations-list row trimming', () => { + const listTool = GENERATED_TOOLS['vision-scanners-observations-list'] as () => ToolBase + + it('drops the repeated snapshot fields and the duplicated reasoning segments', async () => { + const result: any = await listTool().handler(mockContext(), { scanner_id: 'scanner-1' }) + const [row] = result.results + + expect(row.scanner_snapshot.scanner_config).not.toHaveProperty('prompt') + expect(row.scanner_snapshot).not.toHaveProperty('query') + expect(row.scanner_result.model_output).not.toHaveProperty('reasoning_segments') + }) + + it('keeps the per-row output a caller reads, and the version tag beside it', async () => { + const result: any = await listTool().handler(mockContext(), { scanner_id: 'scanner-1' }) + const [row] = result.results + + expect(row.scanner_result.model_output.reasoning).toBe('The shopper opened the cart and left without paying.') + expect(row.scanner_result.model_output.verdict).toBe('yes') + expect(row.scanner_snapshot.scanner_version).toBe(3) + // Trimming one config key must not take the rest of the config with it. + expect(row.scanner_snapshot.scanner_config.allow_inconclusive).toBe(true) + // Enrichment runs on the trimmed rows, so each row still deep-links its recording. + expect(row._posthogUrl).toBe('https://us.posthog.com/project/1/replay/sess-1') + }) + + it('defaults to a page a caller can read without naming a limit', () => { + expect(listTool().schema.parse({ scanner_id: 'scanner-1' }).limit).toBe(20) + }) +}) From db591a8e6420280efe54a8037143f30f7eac74a8 Mon Sep 17 00:00:00 2001 From: "tests-posthog[bot]" <250237707+tests-posthog[bot]@users.noreply.github.com> Date: Sun, 6 Sep 2026 16:01:16 +0000 Subject: [PATCH 2/7] chore: update OpenAPI generated types --- products/replay_vision/mcp/tools.yaml | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/products/replay_vision/mcp/tools.yaml b/products/replay_vision/mcp/tools.yaml index 81a435fdd054..b5066be3181c 100644 --- a/products/replay_vision/mcp/tools.yaml +++ b/products/replay_vision/mcp/tools.yaml @@ -450,29 +450,28 @@ tools: failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has - found over time; to see every scanner's observations for one session, use `vision-observations-list`. - Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and - `scanner_snapshot.query` are the same on every row, and `model_output.reasoning_segments` restates the - flat `reasoning`, so none of the three are on a row. Call `vision-scanners-observations-get` for one - observation's full prompt and citation segments, or `vision-scanners-get` for the scanner's current - prompt. + found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows + are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` are + the same on every row, and `model_output.reasoning_segments` restates the flat `reasoning`, so none of the + three are on a row. Call `vision-scanners-observations-get` for one observation's full prompt and citation + segments, or `vision-scanners-get` for the scanner's current prompt. + agent_note: > + Each observation's `_posthogUrl` opens the recording it analysed. This list carries no citation timestamps. + When you report a finding to someone, deep-link the one or two moments it turns on rather than only + describing them: read that observation with `vision-scanners-observations-get`, take a `chip` segment's + `timestamp_ms` from `scanner_result.model_output.reasoning_segments`, and append `?t=` + (`timestamp_ms` / 1000, rounded down) to the recording URL. param_overrides: limit: description: Number of observations to return per page (default 20). default: 20 + ui_app: vision-observation-list + url_prefix: /replay response: exclude: - scanner_snapshot.scanner_config.prompt - scanner_snapshot.query - scanner_result.model_output.reasoning_segments - agent_note: > - Each observation's `_posthogUrl` opens the recording it analysed. This list carries no citation - timestamps. When you report a finding to someone, deep-link the one or two moments it turns on rather - than only describing them: read that observation with `vision-scanners-observations-get`, take a `chip` - segment's `timestamp_ms` from `scanner_result.model_output.reasoning_segments`, and append - `?t=` (`timestamp_ms` / 1000, rounded down) to the recording URL. - ui_app: vision-observation-list - url_prefix: /replay vision-scanners-observations-retry-create: operation: vision_scanners_observations_retry_create enabled: false From 61fd5bdcb8405116835cad722b7b7c42a9a8ccb9 Mon Sep 17 00:00:00 2001 From: "posthog[bot]" <206114724+posthog[bot]@users.noreply.github.com> Date: Sun, 6 Sep 2026 16:25:11 +0000 Subject: [PATCH 3/7] fix(replay-vision): name the right citation field per scanner type The observations-list agent note told every caller to read citation timestamps from `reasoning_segments`. Summarizer output has no such field; its citation segments are `summary_segments`. The note now names the field for each scanner type. It also claimed the list carries no citation timestamps. That is false for summarizer rows, whose `summary_segments` the new exclude path does not match, so the claim is dropped. Regenerated `services/mcp/src/tools/generated/replay_vision.ts` from `tools.yaml`. Generated-By: PostHog Desktop Task-Id: 3458d9f4-4d0e-4a22-a7d3-1386f65d3e47 --- products/replay_vision/mcp/tools.yaml | 11 ++++++----- services/mcp/src/tools/generated/replay_vision.ts | 2 +- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/products/replay_vision/mcp/tools.yaml b/products/replay_vision/mcp/tools.yaml index b5066be3181c..38fca2bc403c 100644 --- a/products/replay_vision/mcp/tools.yaml +++ b/products/replay_vision/mcp/tools.yaml @@ -456,11 +456,12 @@ tools: three are on a row. Call `vision-scanners-observations-get` for one observation's full prompt and citation segments, or `vision-scanners-get` for the scanner's current prompt. agent_note: > - Each observation's `_posthogUrl` opens the recording it analysed. This list carries no citation timestamps. - When you report a finding to someone, deep-link the one or two moments it turns on rather than only - describing them: read that observation with `vision-scanners-observations-get`, take a `chip` segment's - `timestamp_ms` from `scanner_result.model_output.reasoning_segments`, and append `?t=` - (`timestamp_ms` / 1000, rounded down) to the recording URL. + Each observation's `_posthogUrl` opens the recording it analysed. When you report a finding to someone, + deep-link the one or two moments it turns on rather than only describing them: read that observation with + `vision-scanners-observations-get`, take a `chip` segment's `timestamp_ms` from its citation segments, and + append `?t=` (`timestamp_ms` / 1000, rounded down) to the recording URL. The citation segments are + `scanner_result.model_output.summary_segments` on a summarizer observation, and + `scanner_result.model_output.reasoning_segments` on a monitor, classifier, or scorer one. param_overrides: limit: description: Number of observations to return per page (default 20). diff --git a/services/mcp/src/tools/generated/replay_vision.ts b/services/mcp/src/tools/generated/replay_vision.ts index ee192f21ef6c..4319a2a1e61a 100644 --- a/services/mcp/src/tools/generated/replay_vision.ts +++ b/services/mcp/src/tools/generated/replay_vision.ts @@ -584,7 +584,7 @@ const visionScannersObservationsList = (): ToolBase< }, '/replay' ), - "Each observation's `_posthogUrl` opens the recording it analysed. This list carries no citation timestamps. When you report a finding to someone, deep-link the one or two moments it turns on rather than only describing them: read that observation with `vision-scanners-observations-get`, take a `chip` segment's `timestamp_ms` from `scanner_result.model_output.reasoning_segments`, and append `?t=` (`timestamp_ms` / 1000, rounded down) to the recording URL.\n" + "Each observation's `_posthogUrl` opens the recording it analysed. When you report a finding to someone, deep-link the one or two moments it turns on rather than only describing them: read that observation with `vision-scanners-observations-get`, take a `chip` segment's `timestamp_ms` from its citation segments, and append `?t=` (`timestamp_ms` / 1000, rounded down) to the recording URL. The citation segments are `scanner_result.model_output.summary_segments` on a summarizer observation, and `scanner_result.model_output.reasoning_segments` on a monitor, classifier, or scorer one.\n" ) }, }) From 8ec97e80006d40c41f6c5ebd29b57c4d41c1e49f Mon Sep 17 00:00:00 2001 From: "posthog[bot]" <206114724+posthog[bot]@users.noreply.github.com> Date: Sun, 6 Sep 2026 16:27:46 +0000 Subject: [PATCH 4/7] fix(replay-vision): scope the repeated-field claim to one scanner version The observations-list description said the snapshot prompt and query are the same on every row. They are frozen per observation, and editing either bumps `scanner_version`, so a page written across an edit holds both values. An agent reading that copy attributes older findings to the current prompt. The description now scopes the repetition to one scanner version, says an edit bumps `scanner_version`, and tells a caller spanning versions to read one observation per version before comparing findings. Regenerated the MCP tool schema JSON from `tools.yaml`. Generated-By: PostHog Desktop Task-Id: 3458d9f4-4d0e-4a22-a7d3-1386f65d3e47 --- products/replay_vision/mcp/tools.yaml | 11 +++++++---- services/mcp/schema/generated-tool-definitions.json | 2 +- services/mcp/schema/tool-definitions-all.json | 2 +- 3 files changed, 9 insertions(+), 6 deletions(-) diff --git a/products/replay_vision/mcp/tools.yaml b/products/replay_vision/mcp/tools.yaml index 38fca2bc403c..171737f61fc8 100644 --- a/products/replay_vision/mcp/tools.yaml +++ b/products/replay_vision/mcp/tools.yaml @@ -451,10 +451,13 @@ tools: `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows - are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` are - the same on every row, and `model_output.reasoning_segments` restates the flat `reasoning`, so none of the - three are on a row. Call `vision-scanners-observations-get` for one observation's full prompt and citation - segments, or `vision-scanners-get` for the scanner's current prompt. + are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` + repeat on every row of one scanner version, and `model_output.reasoning_segments` restates the flat + `reasoning`, so none of the three are on a row. Editing a scanner's prompt or query bumps + `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that edit. Call + `vision-scanners-observations-get` for one observation's full prompt, query, and citation segments. When a + page spans more than one `scanner_version`, read one observation per version before you compare findings. + Call `vision-scanners-get` for the scanner's current prompt. agent_note: > Each observation's `_posthogUrl` opens the recording it analysed. When you report a finding to someone, deep-link the one or two moments it turns on rather than only describing them: read that observation with diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json index 315a70e21e32..1a7380a5ce14 100644 --- a/services/mcp/schema/generated-tool-definitions.json +++ b/services/mcp/schema/generated-tool-definitions.json @@ -11673,7 +11673,7 @@ } }, "vision-scanners-observations-list": { - "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` are the same on every row, and `model_output.reasoning_segments` restates the flat `reasoning`, so none of the three are on a row. Call `vision-scanners-observations-get` for one observation's full prompt and citation segments, or `vision-scanners-get` for the scanner's current prompt.", + "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` repeat on every row of one scanner version, and `model_output.reasoning_segments` restates the flat `reasoning`, so none of the three are on a row. Editing a scanner's prompt or query bumps `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that edit. Call `vision-scanners-observations-get` for one observation's full prompt, query, and citation segments. When a page spans more than one `scanner_version`, read one observation per version before you compare findings. Call `vision-scanners-get` for the scanner's current prompt.", "category": "Replay vision", "feature": "replay_vision", "summary": "List scanner observations", diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json index 734e75aef699..ba78b040858c 100644 --- a/services/mcp/schema/tool-definitions-all.json +++ b/services/mcp/schema/tool-definitions-all.json @@ -12301,7 +12301,7 @@ } }, "vision-scanners-observations-list": { - "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` are the same on every row, and `model_output.reasoning_segments` restates the flat `reasoning`, so none of the three are on a row. Call `vision-scanners-observations-get` for one observation's full prompt and citation segments, or `vision-scanners-get` for the scanner's current prompt.", + "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` repeat on every row of one scanner version, and `model_output.reasoning_segments` restates the flat `reasoning`, so none of the three are on a row. Editing a scanner's prompt or query bumps `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that edit. Call `vision-scanners-observations-get` for one observation's full prompt, query, and citation segments. When a page spans more than one `scanner_version`, read one observation per version before you compare findings. Call `vision-scanners-get` for the scanner's current prompt.", "category": "Replay vision", "feature": "replay_vision", "summary": "List scanner observations", From 76ef2c55d3e47690ba41893d2aa5a585be83f5c2 Mon Sep 17 00:00:00 2001 From: "posthog[bot]" <206114724+posthog[bot]@users.noreply.github.com> Date: Sun, 6 Sep 2026 16:32:30 +0000 Subject: [PATCH 5/7] fix(replay-vision): trim summarizer citation segments from list rows too Citation segments are written as `{citation_field}_segments`, and a summarizer cites `summary`, so its duplicate lives in `summary_segments`. The exclude path matched only `reasoning_segments`, so a summarizer page still shipped each body twice. That is the scanner type with the largest free text, so the page-size reduction this change targets missed it. Adds the `summary_segments` exclude path, and a summarizer fixture that proves the row keeps `summary` and loses `summary_segments`. The existing fixtures are all monitor rows, so nothing covered the second field name. The test fails without the exclude path. The description now names the dropped segment field per scanner type. Regenerated the MCP handler and tool schema JSON from `tools.yaml`. Generated-By: PostHog Desktop Task-Id: 3458d9f4-4d0e-4a22-a7d3-1386f65d3e47 --- products/replay_vision/mcp/tools.yaml | 14 ++++--- .../schema/generated-tool-definitions.json | 2 +- services/mcp/schema/tool-definitions-all.json | 2 +- .../mcp/src/tools/generated/replay_vision.ts | 1 + ...scanner-observations-list-trimming.test.ts | 41 ++++++++++++++++++- 5 files changed, 50 insertions(+), 10 deletions(-) diff --git a/products/replay_vision/mcp/tools.yaml b/products/replay_vision/mcp/tools.yaml index 171737f61fc8..ebb31dde69a7 100644 --- a/products/replay_vision/mcp/tools.yaml +++ b/products/replay_vision/mcp/tools.yaml @@ -452,12 +452,13 @@ tools: scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` - repeat on every row of one scanner version, and `model_output.reasoning_segments` restates the flat - `reasoning`, so none of the three are on a row. Editing a scanner's prompt or query bumps - `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that edit. Call - `vision-scanners-observations-get` for one observation's full prompt, query, and citation segments. When a - page spans more than one `scanner_version`, read one observation per version before you compare findings. - Call `vision-scanners-get` for the scanner's current prompt. + repeat on every row of one scanner version, so neither is on a row. The citation segments restate the flat + text they were split from and are dropped too: `model_output.summary_segments` on a summarizer + observation, `model_output.reasoning_segments` on a monitor, classifier, or scorer one. Editing a scanner's + prompt or query bumps `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that + edit. Call `vision-scanners-observations-get` for one observation's full prompt, query, and citation + segments. When a page spans more than one `scanner_version`, read one observation per version before you + compare findings. Call `vision-scanners-get` for the scanner's current prompt. agent_note: > Each observation's `_posthogUrl` opens the recording it analysed. When you report a finding to someone, deep-link the one or two moments it turns on rather than only describing them: read that observation with @@ -476,6 +477,7 @@ tools: - scanner_snapshot.scanner_config.prompt - scanner_snapshot.query - scanner_result.model_output.reasoning_segments + - scanner_result.model_output.summary_segments vision-scanners-observations-retry-create: operation: vision_scanners_observations_retry_create enabled: false diff --git a/services/mcp/schema/generated-tool-definitions.json b/services/mcp/schema/generated-tool-definitions.json index 1a7380a5ce14..3f4a3bcc3c2f 100644 --- a/services/mcp/schema/generated-tool-definitions.json +++ b/services/mcp/schema/generated-tool-definitions.json @@ -11673,7 +11673,7 @@ } }, "vision-scanners-observations-list": { - "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` repeat on every row of one scanner version, and `model_output.reasoning_segments` restates the flat `reasoning`, so none of the three are on a row. Editing a scanner's prompt or query bumps `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that edit. Call `vision-scanners-observations-get` for one observation's full prompt, query, and citation segments. When a page spans more than one `scanner_version`, read one observation per version before you compare findings. Call `vision-scanners-get` for the scanner's current prompt.", + "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` repeat on every row of one scanner version, so neither is on a row. The citation segments restate the flat text they were split from and are dropped too: `model_output.summary_segments` on a summarizer observation, `model_output.reasoning_segments` on a monitor, classifier, or scorer one. Editing a scanner's prompt or query bumps `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that edit. Call `vision-scanners-observations-get` for one observation's full prompt, query, and citation segments. When a page spans more than one `scanner_version`, read one observation per version before you compare findings. Call `vision-scanners-get` for the scanner's current prompt.", "category": "Replay vision", "feature": "replay_vision", "summary": "List scanner observations", diff --git a/services/mcp/schema/tool-definitions-all.json b/services/mcp/schema/tool-definitions-all.json index ba78b040858c..43cc56a8e700 100644 --- a/services/mcp/schema/tool-definitions-all.json +++ b/services/mcp/schema/tool-definitions-all.json @@ -12301,7 +12301,7 @@ } }, "vision-scanners-observations-list": { - "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` repeat on every row of one scanner version, and `model_output.reasoning_segments` restates the flat `reasoning`, so none of the three are on a row. Editing a scanner's prompt or query bumps `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that edit. Call `vision-scanners-observations-get` for one observation's full prompt, query, and citation segments. When a page spans more than one `scanner_version`, read one observation per version before you compare findings. Call `vision-scanners-get` for the scanner's current prompt.", + "description": "List the observations recorded by a single scanner. Each observation is one scan of one session and includes its `status` (pending/running/succeeded/failed/ineligible), `error_reason` (set on terminal non-success statuses, e.g. `too_short`/`no_recording` for ineligible or `provider_rejected`/`validation_failed` for failed — useful for triaging why a scan produced no result), `triggered_by` (`schedule` or `on_demand`), `workflow_id`, the `scanner_result` (the LLM output validated against the scanner's response schema for that scanner_type), and the `session_id` of the recording it analysed. Use this to inspect what a scanner has found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` repeat on every row of one scanner version, so neither is on a row. The citation segments restate the flat text they were split from and are dropped too: `model_output.summary_segments` on a summarizer observation, `model_output.reasoning_segments` on a monitor, classifier, or scorer one. Editing a scanner's prompt or query bumps `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that edit. Call `vision-scanners-observations-get` for one observation's full prompt, query, and citation segments. When a page spans more than one `scanner_version`, read one observation per version before you compare findings. Call `vision-scanners-get` for the scanner's current prompt.", "category": "Replay vision", "feature": "replay_vision", "summary": "List scanner observations", diff --git a/services/mcp/src/tools/generated/replay_vision.ts b/services/mcp/src/tools/generated/replay_vision.ts index 4319a2a1e61a..acbb6b840a16 100644 --- a/services/mcp/src/tools/generated/replay_vision.ts +++ b/services/mcp/src/tools/generated/replay_vision.ts @@ -568,6 +568,7 @@ const visionScannersObservationsList = (): ToolBase< 'scanner_snapshot.scanner_config.prompt', 'scanner_snapshot.query', 'scanner_result.model_output.reasoning_segments', + 'scanner_result.model_output.summary_segments', ]) ), } as typeof result diff --git a/services/mcp/tests/unit/vision-scanner-observations-list-trimming.test.ts b/services/mcp/tests/unit/vision-scanner-observations-list-trimming.test.ts index c0108af69680..8a72ffd22252 100644 --- a/services/mcp/tests/unit/vision-scanner-observations-list-trimming.test.ts +++ b/services/mcp/tests/unit/vision-scanner-observations-list-trimming.test.ts @@ -32,12 +32,39 @@ const FULL_ROW = { }, } -function mockContext(): Context { +// A summarizer row: same duplication, under `summary_segments`, because segments are written as +// `{citation_field}_segments` and summarizer cites `summary` rather than `reasoning`. +const SUMMARIZER_ROW = { + id: 'obs-2', + session_id: 'sess-2', + status: 'succeeded', + scanner_snapshot: { + name: 'Session summaries', + scanner_type: 'summarizer', + scanner_version: 1, + scanner_config: { prompt: 'Summarize what the person did.', length: 'long' }, + query: { kind: 'RecordingsQuery', events: [{ id: '$pageview' }] }, + }, + scanner_result: { + model_output: { + scanner_type: 'summarizer', + title: 'Renamed a project, then opened billing', + summary: 'The person renamed a project and then opened the billing page.', + summary_segments: [ + { kind: 'text', value: 'The person renamed a project and then opened the billing page.' }, + { kind: 'chip', timestamp_ms: 91000 }, + ], + }, + signals_count: 0, + }, +} + +function mockContext(row: object = FULL_ROW): Context { return { stateManager: { getProjectId: async () => 1 }, api: { getProjectBaseUrl: () => 'https://us.posthog.com/project/1', - request: async () => ({ count: 1, next: null, previous: null, results: [structuredClone(FULL_ROW)] }), + request: async () => ({ count: 1, next: null, previous: null, results: [structuredClone(row)] }), }, } as unknown as Context } @@ -67,6 +94,16 @@ describe('vision-scanners-observations-list row trimming', () => { expect(row._posthogUrl).toBe('https://us.posthog.com/project/1/replay/sess-1') }) + it('drops the duplicated summary segments on a summarizer row, and keeps the summary', async () => { + const result: any = await listTool().handler(mockContext(SUMMARIZER_ROW), { scanner_id: 'scanner-2' }) + const [row] = result.results + + expect(row.scanner_result.model_output).not.toHaveProperty('summary_segments') + expect(row.scanner_result.model_output.summary).toBe( + 'The person renamed a project and then opened the billing page.' + ) + }) + it('defaults to a page a caller can read without naming a limit', () => { expect(listTool().schema.parse({ scanner_id: 'scanner-1' }).limit).toBe(20) }) From b71ecca68777be9809a628be14413aa5c5a9031b Mon Sep 17 00:00:00 2001 From: "tests-posthog[bot]" <250237707+tests-posthog[bot]@users.noreply.github.com> Date: Sun, 6 Sep 2026 16:36:09 +0000 Subject: [PATCH 6/7] chore: update OpenAPI generated types --- products/replay_vision/mcp/tools.yaml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/products/replay_vision/mcp/tools.yaml b/products/replay_vision/mcp/tools.yaml index ebb31dde69a7..ec4a57e9e96f 100644 --- a/products/replay_vision/mcp/tools.yaml +++ b/products/replay_vision/mcp/tools.yaml @@ -453,12 +453,12 @@ tools: found over time; to see every scanner's observations for one session, use `vision-observations-list`. Rows are trimmed to keep a page small. `scanner_snapshot.scanner_config.prompt` and `scanner_snapshot.query` repeat on every row of one scanner version, so neither is on a row. The citation segments restate the flat - text they were split from and are dropped too: `model_output.summary_segments` on a summarizer - observation, `model_output.reasoning_segments` on a monitor, classifier, or scorer one. Editing a scanner's - prompt or query bumps `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that - edit. Call `vision-scanners-observations-get` for one observation's full prompt, query, and citation - segments. When a page spans more than one `scanner_version`, read one observation per version before you - compare findings. Call `vision-scanners-get` for the scanner's current prompt. + text they were split from and are dropped too: `model_output.summary_segments` on a summarizer observation, + `model_output.reasoning_segments` on a monitor, classifier, or scorer one. Editing a scanner's prompt or + query bumps `scanner_snapshot.scanner_version`, and a page can hold rows from either side of that edit. Call + `vision-scanners-observations-get` for one observation's full prompt, query, and citation segments. When a + page spans more than one `scanner_version`, read one observation per version before you compare findings. + Call `vision-scanners-get` for the scanner's current prompt. agent_note: > Each observation's `_posthogUrl` opens the recording it analysed. When you report a finding to someone, deep-link the one or two moments it turns on rather than only describing them: read that observation with From b7baa88d29c799e1aae0b9998eb3372ee3600f25 Mon Sep 17 00:00:00 2001 From: "posthog[bot]" <206114724+posthog[bot]@users.noreply.github.com> Date: Sun, 6 Sep 2026 16:36:44 +0000 Subject: [PATCH 7/7] docs(replay-vision): route the skill's citation workflow through retrieve The bundled `exploring-replay-vision-observations` skill names `vision-scanners-observations-list` as the workhorse, then reads citation segments straight off those rows. Trimming the segments off that page leaves the skill prescribing a field the tool no longer returns, so an agent following it drops the deep links the section calls the point of the step. The section now says the scanner-scoped list leaves the segments off, and to read the observation with `vision-scanners-observations-get` before citing it. It also names the segment field per scanner type, and records that the session-scoped tools still return segments, so that path needs no extra call. The skill is seeded per team as an external companion skill, so scout runs load this text. Generated-By: PostHog Desktop Task-Id: 3458d9f4-4d0e-4a22-a7d3-1386f65d3e47 --- .../skills/exploring-replay-vision-observations/SKILL.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/products/replay_vision/skills/exploring-replay-vision-observations/SKILL.md b/products/replay_vision/skills/exploring-replay-vision-observations/SKILL.md index 036d98030954..2da774d3f7c6 100644 --- a/products/replay_vision/skills/exploring-replay-vision-observations/SKILL.md +++ b/products/replay_vision/skills/exploring-replay-vision-observations/SKILL.md @@ -99,10 +99,14 @@ and, like all observations, runs at most once per `(scanner, session)`. ### Cite moments, not just sessions -`scanner_result.model_output.reasoning_segments` is the same prose as `reasoning`, pre-split into `text` segments and `chip` segments. +An observation's citation segments are the same prose as its flat text, pre-split into `text` segments and `chip` segments. +The field is `scanner_result.model_output.summary_segments` on a summarizer, and `scanner_result.model_output.reasoning_segments` on a monitor, classifier, or scorer. Each chip carries a `timestamp_ms`: the recording-relative offset of the moment the model is pointing at. That's what makes a finding checkable — it turns "the user hit a paywall" into a link that opens on the paywall. +`vision-scanners-observations-list` leaves the segments off its rows to keep a page small, so read the observation you want to cite with `vision-scanners-observations-get` first. +`vision-observations-list` and `vision-observations-retrieve` still return them, so a session-scoped page needs no extra call. + The observation's `_posthogUrl` is its recording; append `?t=` (`timestamp_ms` / 1000, rounded down) to seek there. ```text