Skip to content

perf(replay): draw screenshots in the screen's color space and convert off the main thread - #1028

Merged
ioannisj merged 3 commits into
PostHog:mainfrom
Jaltaire:perf/replay-screenshot-color-space
Nov 11, 2026
Merged

ioannisj merged 3 commits into
PostHog:mainfrom
Jaltaire:perf/replay-screenshot-color-space

Conversation

@Jaltaire

@Jaltaire Jaltaire commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

drawHierarchy hands back the render server's frame in the screen's own color space, which is Display P3 on every wide-gamut iPhone. PostHogGraphicsImageRenderer always allocates a device RGB bitmap, so drawHierarchy converts the whole frame, pixel by pixel, on the main thread while it holds it.

I found this while profiling screenshot replay in an app whose UI is a single full-screen web view playing a WebGL animation. In a Time Profiler trace on an iOS 27 simulator, 187 of the 197 ms of main-thread CPU that replay used over 15 s were in CGContextDrawImage's color conversion, called from drawHierarchy.

What changes:

  • The two drawHierarchy paths in UIView.toImage (fidelity and afterScreenUpdates) draw into a bitmap in the screen's gamut: Display P3 when screen.traitCollection.displayGamut is .P3, device RGB otherwise. The draw becomes a copy. One drawsHierarchy flag picks both the gamut and the renderer, so the two can't disagree.
  • The frame is converted off the main thread, on the replay queue, and where it lands depends on the path:
    • Resized or masked: the encode-time resample and mask redraw in RRWireframe already draw into device RGB, so they convert to device RGB as they draw, as on main.
    • Native resolution with nothing to mask (screenshotScale = 1.0): resample returns the image as is, so it reaches the WebP encoder in Display P3, and the encoder now converts a wide-gamut image to sRGB.
    • JPEG fallback (only when WebP encoding fails): no conversion. It encodes the image in the color space it arrives in, so a native-resolution, unmasked screenshot goes out in Display P3 rather than device RGB.
    • An image in device RGB or sRGB encodes exactly as before, which the existing WebP golden tests check byte for byte.
  • The WebP encoder is shared, so the sRGB conversion also reaches images that skip the redraw outside screenshots: wireframe-mode UIImageView images (image.toBase64() in RRWireframe.toDict()) and the deprecated public imageToBase64. A Display P3 image there used to upload its raw P3 values, which decoders read as sRGB and show desaturated. It now uploads converted to sRGB, in its true colors. For a P3 (0.8, 0.2, 0.2) fill, review measured main decoding as [187, 66, 59] and this branch as [205, 52, 51].
  • The presentation-tree renderer (layer.render(in:)) keeps its device RGB bitmap. It draws layer contents, which are mostly sRGB already, so a P3 bitmap would add a conversion there rather than remove one.

No public API change. Screenshot colors are unchanged, within WebP's rounding, except in the JPEG fallback above. Wide-gamut images in wireframe mode and from imageToBase64 now upload in sRGB, as described above.

💚 How did you test it?

Measurement. PostHogExample, Release, on an iPhone Air simulator (iOS 27.0, 420×912 pt at 3x, Display P3), with a local, uncommitted patch that:

  • shows one full-screen WKWebView playing a WebGL shader;
  • turns on screenshot replay (masking off) against a local mock server;
  • asks the key window for a layout pass every second, so replay captures about once a second;
  • times toImage and counts frames a CADisplayLink missed.

I ran main (29d7d86) and this branch interleaved, 60 s per run:

Run toImage on main, median (p10–p90) Frames dropped per capture, mean Frames kept
main, round 1 37.1 ms (34.7–40.9) 2.19 52
this PR, round 1 22.6 ms (20.0–26.4) 0.94 52
main, round 2 39.1 ms (36.9–42.3) 2.33 52
this PR, round 2 21.9 ms (20.5–26.6) 1.23 50
main, round 3 (run later) 37.3 ms (34.4–44.3) 2.40 53
this PR, round 3 (run later) 22.5 ms (18.7–27.9) 1.13 55

So each capture holds the main thread for about 22 ms instead of 38 ms, and costs about one frame at 60 Hz instead of two or three. Frames kept are the same, and so are the uploads: 420×912 at one pixel per point, as on main. The ~22 ms that remains is mostly drawHierarchy's own render-server snapshot, which this PR doesn't touch. Frames were counted 150 ms around each render with a CADisplayLink, after a 5 s settle, on a simulator, so absolute numbers on a device will differ, while the conversion this PR removes is per pixel on any wide-gamut screen.

Tests. New Swift Testing suites, all on an iOS 27.0 simulator:

  • Replay screenshot color space (in PostHogGraphicsImageRendererTest.swift):
    • a P3 gamut gives a Display P3 bitmap, while sRGB, unspecified, and the default give device RGB;
    • drawHierarchy uses the screen's gamut, and the presentation tree and a view outside a window use device RGB;
    • a presentation-tree screenshot stays device RGB;
    • a Display P3 screenshot uploads in its sRGB colors when it is resampled, at native resolution (screenshotScale = 1.0, no redraw), and around a mask.
  • WebP color space (in PostHogWebPTest.swift): the encoder's color space for P3, sRGB, device RGB, gray, and missing color spaces, and an end-to-end encode of a P3 image.
  • Replay screenshot failed draws (existing) now also checks that afterScreenUpdates: true draws the hierarchy when preferFidelityRenderer is false.

With the encoder change reverted, the native-resolution upload test and the P3 encode test fail, and the rest of both suites still pass.

Validation run:

  • Full iOS simulator suite (scripts/test-ios-simulator.sh's steps, on an iPhone Air with iOS 27.0): 1,671 tests in 232 suites passed, with the existing known issue in PostHogMaskFlattenedContentTest.
  • make testPresentationMasks suites, app-hosted, where drawHierarchy really renders: 31 tests in 2 suites passed.
  • make test (SPM, macOS): 1,170 tests in 177 suites passed.
  • make lint with the CI-pinned SwiftFormat 0.62.1: clean. SwiftLint reports only the existing warnings, none in changed files.
  • make apiCheck: the public API snapshot is unchanged.
  • make checkTestRegistration: all 109 test files are registered, and no new test file was added.
  • make buildSdk targets (iOS, macOS, Mac Catalyst, tvOS, watchOS, visionOS, and swift build): all built.
  • Not run: make maskSnapshots, because the pinned Xcode 26.3 and iOS 26.2 runtime aren't installed here. The golden harness renders with its own layer.render(in:) and only shares RRWireframe.maskImage, whose device RGB redraw is unchanged, so I expect no golden to move. CI's run is the check.
  • Toolchain note: the only Xcode here is 27.2, which rejects the project's iOS 13, macOS 10.15, tvOS 13, and watchOS 6 deployment targets. Every local build above passed IPHONEOS_DEPLOYMENT_TARGET=15.0 and the matching floors on the command line. The project files are untouched.

After review, on this branch merged with current main (d237d54):

  • Full iOS simulator suite (scripts/test-ios-simulator.sh's steps, on an iPhone Air with iOS 27.0): 1,726 tests in 242 suites passed, with the existing known issue in PostHogMaskFlattenedContentTest.
  • make testPresentationMasks suites: 31 tests in 2 suites passed.
  • make test (SPM, macOS): 1,184 tests in 177 suites passed.
  • make swiftFormatCheck with SwiftFormat 0.62.1: clean. make swiftLintCheck: only the existing warnings, none in changed files.
  • make apiCheck: the public API snapshot is up to date.
  • make checkTestRegistration: all 112 test files are registered.

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran pnpm changeset to generate a changeset file (a patch changeset, written in the same format)

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

🤖 Generated with Claude Code

…t off the main thread

drawHierarchy hands back the render server's frame in the screen's own
color space, Display P3 on a wide-gamut screen. The screenshot bitmap
was always device RGB, so every capture converted the whole frame,
pixel by pixel, on the main thread while drawHierarchy held it.

The two drawHierarchy paths now draw into a bitmap in the screen's
gamut, so the draw is a copy. The frame reaches sRGB off the main
thread: the encode-time resample and mask redraw already draw into
device RGB, and the WebP encoder now converts a wide-gamut image to
sRGB, which covers a native-resolution frame with nothing to mask. An
image in device RGB or sRGB encodes exactly as before. The
presentation-tree renderer keeps its device RGB bitmap, since it draws
layer contents that are mostly sRGB already.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ioannisj

Copy link
Copy Markdown
Contributor

Thank you @Jaltaire for this. Taking a look now

@ioannisj ioannisj left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 An AI agent wrote this on behalf of @ioannisj.

The code side reads fine to me, the P3 bitmap either gets redrawn into device RGB by the resample and mask pass in RRWireframe or converted to sRGB inside webpData, so masks land the same and I measured the capture dropping from about 22 ms to 12 ms of main thread per frame on an iPhone 17 Pro. The inline comments are small ones, mostly wording in the changeset and PR body around what actually changes colour, plus one tidy-up in toImage.

Comment thread PostHog/Utils/UIImage+WebP.swift Resolved
Comment thread .changeset/wide-gamut-screens-copy.md Resolved
Comment thread PostHog/Replay/UIView+Util.swift Resolved
Jaltaire and others added 2 commits November 10, 2026 13:18
…wsHierarchy flag

toImage computed the bitmap's gamut from afterScreenUpdates ||
preferFidelityRenderer, then branched the renderer on the same two
flags separately, so the rule that only the drawHierarchy paths get the
screen's gamut lived in two places that had to agree. Both now read a
single drawsHierarchy flag, and the two drawHierarchy branches become
one call that passes afterScreenUpdates through.

The failed-draw test now also covers afterScreenUpdates without the
fidelity renderer, which must still draw the hierarchy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The changeset said every screenshot reaches sRGB when it is resized,
masked, or encoded. A resized or masked screenshot is redrawn in device
RGB, only the WebP encoder targets sRGB, and the JPEG fallback does not
convert. It also left out that the shared WebP encoder now converts
wide-gamut images in wireframe mode and from imageToBase64 to sRGB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Jaltaire

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @ioannisj. I pushed two commits:

  • webPColorSpace in the shared encoder: I kept the conversion where it is, since the sRGB output is the correct one. The changeset and the PR body now say that wide-gamut images in wireframe mode and from imageToBase64 are converted to sRGB too.
  • Changeset wording: it now names each path. A resized or masked screenshot is redrawn in device RGB, an image that reaches the WebP encoder in a wide gamut (such as a native-resolution screenshot with nothing to mask) is converted to sRGB, and the JPEG fallback doesn't convert. The PR body spells out the same paths.
  • drawsHierarchy: toImage computes let drawsHierarchy = afterScreenUpdates || preferFidelityRenderer once, and both the gamut and the renderer branch on it. The two drawHierarchy branches are now one call that passes afterScreenUpdates through, and the failed-draw test also covers afterScreenUpdates: true with preferFidelityRenderer: false.

On the test-ios-simulator (unit tests) failure: it's PostHogPushNotificationTest "config.optOut before setup: one DELETE, the record parked, nothing on a later flush", which saw 2 requests instead of 1. I think the test is racing the client. It calls flush() as soon as the mock server has recorded the setup DELETE, but the pending-unregister intent is only cleared once the client handles that DELETE's 2xx. A flush() in that gap passes the stillPending check in attemptUnregister and queues a second DELETE behind the first. On main, with only the stubbed DELETE response delayed by 0.2 s, the test fails at the same assertion. This PR doesn't touch push code, and the test passed 30 of 30 repetitions here.

@ioannisj
ioannisj enabled auto-merge (squash) November 11, 2026 09:12
@ioannisj
ioannisj merged commit 32f78c6 into PostHog:main Nov 11, 2026
52 of 53 checks passed
@Jaltaire
Jaltaire deleted the perf/replay-screenshot-color-space branch November 11, 2026 19:10
turnipdabeets added a commit that referenced this pull request Nov 12, 2026
RRWireframe only builds with SessionReplay in 4.0, so the no-traits test build failed on the tests main added in #1028.
turnipdabeets added a commit that referenced this pull request Nov 12, 2026
4.0 renamed RRWireframe to RRScreenshotFrame, so the iOS test build failed on the tests from #1028. This replaces the trait gating from the previous two commits, which was the wrong fix: the type is not trait-gated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants