Skip to content

docs(compliance): weekly SDK compliance matrix - #148

Draft
posthog[bot] wants to merge 3 commits into
mainfrom
posthog-code/weekly-sdk-compliance-matrix
Draft

posthog[bot] wants to merge 3 commits into
mainfrom
posthog-code/weekly-sdk-compliance-matrix

Conversation

@posthog

@posthog posthog Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Weekly SDK compliance matrix

The living per-SDK compliance matrix under compliance/, maintained by a bounded weekly audit:
≤3 SDKs are deeply verified per run (spec-affected → code-changed → pending-initial-audit →
staleness-backstop), so every SDK is re-checked roughly every 4 weeks rather than all at once.

Why: the cross-SDK contracts in openspec/specs/ are the point of this repo, but nothing
recorded how far each SDK actually is from them. This gives that a durable, per-contract home with
evidence and a backwards-compatibility verdict attached to every gap, so the specs can be used to
plan SDK work instead of only to settle arguments.

Scope: compliance/ only. No spec content, no SDK code, nothing outside that directory.


This run (2026-11-09) — posthog-php, posthog-ruby, posthog-go

The contract surface grew from 68 to 69: Experiment Exposure
(#162) — for a config version 2 flag, a read landing
on an experiment split emits $experiment_exposure directly. sdk-specs advanced 19 commits over
23 spec files.

Selection. posthog-php and posthog-ruby were the two stalest pointers (6 weeks, the only two
outside the ~4-week window) and had the largest backfill — they were the top of last run's queue.
posthog-go is the only remaining matrix SDK in the mcp-analytics applicability table, which five of
this cycle's commits touch. All three are server SDKs, which is where the Emails, Exception Event
Metadata, Tracing Headers and config-version-2 flag deltas land. All three had also drifted hard
(+3,409, +1,695 and +11,341 non-test lines respectively).

Highlights

  • Emails went from one implementation to four. It was a ❌ on posthog-java three weeks ago. All
    three SDKs audited this run ship it and all three pass — the flattened template_id body from
    #163, Cloud app-host resolution with a custom-host
    refusal, and an idempotency key built once per call outside the attempt loop. posthog-ruby
    reproduces the spec's error-name table line for line.
  • The OpenTelemetry bridge landed in all three, independently, within weeks.
    #129's span enrichment and
    #123's trace context on exception events are
    implemented in each, with the same shape. posthog-go alone gets the detached-work scenario
    right
    (an active flag the middleware clears with defer), and posthog-go alone has the
    trace-context path reachable only through an opt-in helper
    in a separate module — a Go design
    question rather than a bug, recorded as such.
  • None of the three reads /flags?v=4. Experiment Exposure is therefore ➖ on all three by the
    spec's own applicability sentence, and the config-version-2 attribution requirements never
    trigger. Raised as a scope question rather than scored as a gap.
  • posthog-ruby ships MCP analytics; the spec says it does not. lib/posthog/mcp/ is 3,798 lines
    across 22 files and grew again this cycle, while the applicability table records posthog-ruby as
    "not yet". Scored 🟡; the table is flagged for a human.
  • The matrix still has no Unknown cells.

Gaps discovered this run

SDK Contract Status Backwards-compat
posthog-php Retry Queue — no jitter (was ✅) 🟡 Backward-compatible
posthog-ruby MCP Analytics — $mcp_interface missing, no client conversation ids 🟡 Backward-compatible
posthog-go Retry Queue — jitter: 0 at every ingestion call site 🟡 Backward-compatible
posthog-go MCP Analytics — $mcp_interface missing, no client conversation ids 🟡 Backward-compatible

The jitter requirement (#132, SHOULD-tier) is now
unmet on six of the seven SDKs checked against it. posthog-go is the sharpest case: NewBackoff
supports jitter and every ingestion and flag call site passes 0, while its own new Emails client
passes 0.2. posthog-php has the same split. posthog-ruby is the one that passes, and is the
reference.

Gaps resolved this run

  • Emails ✅ on all three rather than the ❌ posthog-java's row set the precedent for.
  • posthog-php's and posthog-ruby's trace-context and span-bridge requirements, both added this
    cycle, are satisfied on arrival — neither became a gap.
  • #169's evaluation_runtime: "client" warning
    exclusion is implemented in all three, five days after it merged.
  • No previously recorded ❌ was found fixed, and no new ❌ was added.

Needs a human

  1. The mcp-analytics applicability table is stale in two places. It records posthog-ruby as
    "not yet" against a 3,798-line shipped package, and lists two Go adapter packages where posthog-go
    now ships three (posthogmcpgo, 1,137 lines, new this cycle). The ruby entry changed a row's
    status: this run scored the package anyway and gave it 🟡, which a maintainer may want reversed
    to ➖ until the table is updated.
  2. Does a server SDK that stays on /flags?v=2 keep scoring ➖ on Experiment Exposure
    indefinitely?
    Three server SDKs cannot produce correct experiment analytics for a config
    version 2 flag, and the matrix shows no gap for it, because the applicability sentence sanctions
    abstaining. If v=4 adoption is expected, that sentence is the place to say so.
  3. posthog-rs is still an open scope question, carried from the previous run.
  4. posthog-java's Traces row still needs a direction, carried from the previous run.

Running state across all runs

Audited at least once: all 12 in-scope SDKs. Pending initial audit: none.

SDK Last audited Rows
posthog-php 2026-11-09 · 9c059270 69
posthog-ruby 2026-11-09 · c08fd1b9 69
posthog-go 2026-11-09 · 2f949f18 69
posthog-java 2026-11-02 · 8309f837 68 (+1 needed)
posthog-node 2026-11-02 · 8c832754 68 (+1 needed)
posthog-python 2026-11-02 · dca22075 68 (+1 needed)
posthog-android 2026-10-19 · c5e788bd 64 (+5 needed)
posthog-flutter 2026-10-19 · 1032c102 64 (+5 needed)
posthog-react-native 2026-10-19 · 3ddaa654 64 (+5 needed)
posthog-ios 2026-10-08 · f5cbe87c 64 (+5 needed)
posthog-js 2026-10-05 · 6538babc 64 (+5 needed)
posthog-dotnet 2026-10-05 · ae4d019c 64 (+5 needed)

Queued for next run, in order: posthog-dotnet and posthog-js (now the stalest at 5 weeks, and the
two whose backfill contains a real scoring question — dotnet's Emails row and js's MCP Analytics
row), then posthog-ios. A half-slot sweep can add the four ➖ rows to the six 64-row files and the
single Experiment Exposure row to java/node/python cheaply — but posthog-js's Experiment Exposure row
should be scored, not swept, since it is both-scoped and js reads /flags.

This PR is the accumulating baseline and is intentionally left as a draft; it is not meant to be
merged mid-rotation.


Created with PostHog Desktop

🤖 Generated with Claude Code

@turnipdabeets
turnipdabeets changed the base branch from docs/acceptance-step-catalogue to main October 27, 2026 22:00
@posthog
posthog Bot force-pushed the posthog-code/weekly-sdk-compliance-matrix branch from 70def4b to 45c47cc Compare November 2, 2026 07:22
posthog Bot added 2 commits November 9, 2026 07:02
Generated-By: PostHog Desktop
Task-Id: 86b4f98b-a338-40c2-912e-9e3a9dc366fd
Deep re-audit of posthog-php, posthog-ruby and posthog-go against the 69
cross-SDK contracts, at sdk-specs 5160f2f.

Selection: php and ruby were the two stalest pointers (6 weeks, outside the
~4-week window) and had the largest backfill; go is the remaining matrix SDK
named in the mcp-analytics applicability table, which five of this cycle's 19
spec commits touch. All three are server SDKs, where the Emails, Exception
Event Metadata, Tracing Headers and config-version-2 flag deltas land.

Contract surface grew to 69 with Experiment Exposure (#162).

Main findings:
- Emails went from one implementation to four. All three SDKs ship it and all
  three pass, including #163's flattened send body, Cloud app-host resolution,
  and a once-per-call idempotency key.
- #129's span-context bridge and #123's trace context on exception events
  landed in all three. posthog-go is alone in getting the detached-work
  scenario right, and alone in the trace-context path being reachable only
  through an opt-in helper in a separate module.
- None of the three reads /flags?v=4, so Experiment Exposure is ➖ on all three
  by the spec's own applicability sentence — raised as a scope question.
- posthog-ruby ships a 3,798-line MCP package the spec's applicability table
  records as "not yet"; scored 🟡, with the table flagged for update.
- The #132 jitter requirement is now unmet on five of seven SDKs checked;
  posthog-go passes jitter: 0 at every ingestion call site while its own
  Emails client jitters.

Scope: compliance/ only.

Generated-By: PostHog Desktop
Task-Id: 86b4f98b-a338-40c2-912e-9e3a9dc366fd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants