Repository navigation
perf(replay): draw screenshots in the screen's color space and convert off the main thread - #1028
Conversation
…t off the main thread drawHierarchy hands back the render server's frame in the screen's own color space, Display P3 on a wide-gamut screen. The screenshot bitmap was always device RGB, so every capture converted the whole frame, pixel by pixel, on the main thread while drawHierarchy held it. The two drawHierarchy paths now draw into a bitmap in the screen's gamut, so the draw is a copy. The frame reaches sRGB off the main thread: the encode-time resample and mask redraw already draw into device RGB, and the WebP encoder now converts a wide-gamut image to sRGB, which covers a native-resolution frame with nothing to mask. An image in device RGB or sRGB encodes exactly as before. The presentation-tree renderer keeps its device RGB bitmap, since it draws layer contents that are mostly sRGB already. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you @Jaltaire for this. Taking a look now |
ioannisj
left a comment
There was a problem hiding this comment.
🤖 An AI agent wrote this on behalf of @ioannisj.
The code side reads fine to me, the P3 bitmap either gets redrawn into device RGB by the resample and mask pass in RRWireframe or converted to sRGB inside webpData, so masks land the same and I measured the capture dropping from about 22 ms to 12 ms of main thread per frame on an iPhone 17 Pro. The inline comments are small ones, mostly wording in the changeset and PR body around what actually changes colour, plus one tidy-up in toImage.
…wsHierarchy flag toImage computed the bitmap's gamut from afterScreenUpdates || preferFidelityRenderer, then branched the renderer on the same two flags separately, so the rule that only the drawHierarchy paths get the screen's gamut lived in two places that had to agree. Both now read a single drawsHierarchy flag, and the two drawHierarchy branches become one call that passes afterScreenUpdates through. The failed-draw test now also covers afterScreenUpdates without the fidelity renderer, which must still draw the hierarchy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The changeset said every screenshot reaches sRGB when it is resized, masked, or encoded. A resized or masked screenshot is redrawn in device RGB, only the WebP encoder targets sRGB, and the JPEG fallback does not convert. It also left out that the shared WebP encoder now converts wide-gamut images in wireframe mode and from imageToBase64 to sRGB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the review, @ioannisj. I pushed two commits:
On the |
RRWireframe only builds with SessionReplay in 4.0, so the no-traits test build failed on the tests main added in #1028.
4.0 renamed RRWireframe to RRScreenshotFrame, so the iOS test build failed on the tests from #1028. This replaces the trait gating from the previous two commits, which was the wrong fix: the type is not trait-gated.
💡 Motivation and Context
drawHierarchyhands back the render server's frame in the screen's own color space, which is Display P3 on every wide-gamut iPhone.PostHogGraphicsImageRendereralways allocates a device RGB bitmap, sodrawHierarchyconverts the whole frame, pixel by pixel, on the main thread while it holds it.I found this while profiling screenshot replay in an app whose UI is a single full-screen web view playing a WebGL animation. In a Time Profiler trace on an iOS 27 simulator, 187 of the 197 ms of main-thread CPU that replay used over 15 s were in
CGContextDrawImage's color conversion, called fromdrawHierarchy.What changes:
drawHierarchypaths inUIView.toImage(fidelity andafterScreenUpdates) draw into a bitmap in the screen's gamut: Display P3 whenscreen.traitCollection.displayGamutis.P3, device RGB otherwise. The draw becomes a copy. OnedrawsHierarchyflag picks both the gamut and the renderer, so the two can't disagree.RRWireframealready draw into device RGB, so they convert to device RGB as they draw, as onmain.screenshotScale = 1.0):resamplereturns the image as is, so it reaches the WebP encoder in Display P3, and the encoder now converts a wide-gamut image to sRGB.UIImageViewimages (image.toBase64()inRRWireframe.toDict()) and the deprecated publicimageToBase64. A Display P3 image there used to upload its raw P3 values, which decoders read as sRGB and show desaturated. It now uploads converted to sRGB, in its true colors. For a P3 (0.8, 0.2, 0.2) fill, review measuredmaindecoding as [187, 66, 59] and this branch as [205, 52, 51].layer.render(in:)) keeps its device RGB bitmap. It draws layer contents, which are mostly sRGB already, so a P3 bitmap would add a conversion there rather than remove one.No public API change. Screenshot colors are unchanged, within WebP's rounding, except in the JPEG fallback above. Wide-gamut images in wireframe mode and from
imageToBase64now upload in sRGB, as described above.💚 How did you test it?
Measurement.
PostHogExample, Release, on an iPhone Air simulator (iOS 27.0, 420×912 pt at 3x, Display P3), with a local, uncommitted patch that:WKWebViewplaying a WebGL shader;toImageand counts frames aCADisplayLinkmissed.I ran
main(29d7d86) and this branch interleaved, 60 s per run:toImageon main, median (p10–p90)main, round 1main, round 2main, round 3 (run later)So each capture holds the main thread for about 22 ms instead of 38 ms, and costs about one frame at 60 Hz instead of two or three. Frames kept are the same, and so are the uploads: 420×912 at one pixel per point, as on
main. The ~22 ms that remains is mostlydrawHierarchy's own render-server snapshot, which this PR doesn't touch. Frames were counted 150 ms around each render with aCADisplayLink, after a 5 s settle, on a simulator, so absolute numbers on a device will differ, while the conversion this PR removes is per pixel on any wide-gamut screen.Tests. New Swift Testing suites, all on an iOS 27.0 simulator:
Replay screenshot color space(inPostHogGraphicsImageRendererTest.swift):drawHierarchyuses the screen's gamut, and the presentation tree and a view outside a window use device RGB;screenshotScale = 1.0, no redraw), and around a mask.WebP color space(inPostHogWebPTest.swift): the encoder's color space for P3, sRGB, device RGB, gray, and missing color spaces, and an end-to-end encode of a P3 image.Replay screenshot failed draws(existing) now also checks thatafterScreenUpdates: truedraws the hierarchy whenpreferFidelityRendereris false.With the encoder change reverted, the native-resolution upload test and the P3 encode test fail, and the rest of both suites still pass.
Validation run:
scripts/test-ios-simulator.sh's steps, on an iPhone Air with iOS 27.0): 1,671 tests in 232 suites passed, with the existing known issue inPostHogMaskFlattenedContentTest.make testPresentationMaskssuites, app-hosted, wheredrawHierarchyreally renders: 31 tests in 2 suites passed.make test(SPM, macOS): 1,170 tests in 177 suites passed.make lintwith the CI-pinned SwiftFormat 0.62.1: clean. SwiftLint reports only the existing warnings, none in changed files.make apiCheck: the public API snapshot is unchanged.make checkTestRegistration: all 109 test files are registered, and no new test file was added.make buildSdktargets (iOS, macOS, Mac Catalyst, tvOS, watchOS, visionOS, andswift build): all built.make maskSnapshots, because the pinned Xcode 26.3 and iOS 26.2 runtime aren't installed here. The golden harness renders with its ownlayer.render(in:)and only sharesRRWireframe.maskImage, whose device RGB redraw is unchanged, so I expect no golden to move. CI's run is the check.IPHONEOS_DEPLOYMENT_TARGET=15.0and the matching floors on the command line. The project files are untouched.After review, on this branch merged with current
main(d237d54):scripts/test-ios-simulator.sh's steps, on an iPhone Air with iOS 27.0): 1,726 tests in 242 suites passed, with the existing known issue inPostHogMaskFlattenedContentTest.make testPresentationMaskssuites: 31 tests in 2 suites passed.make test(SPM, macOS): 1,184 tests in 177 suites passed.make swiftFormatCheckwith SwiftFormat 0.62.1: clean.make swiftLintCheck: only the existing warnings, none in changed files.make apiCheck: the public API snapshot is up to date.make checkTestRegistration: all 112 test files are registered.📝 Checklist
If releasing new changes
pnpm changesetto generate a changeset file (apatchchangeset, written in the same format)🤖 Agent context
Autonomy: Human-driven (agent-assisted)
main, wrote the tests, and took the measurements above.mainthe resample and mask redraw already convert to device RGB off the main thread, so only thedrawHierarchybitmap and the WebP encoder needed changing.drawHierarchyreturns the render server's frame in the display's color space whatever a view's trait overrides say.🤖 Generated with Claude Code