cache: shared-memory-backed Dir for fast restart by masaori335 · Pull Request #13328 · apache/trafficserver

masaori335 · 2026-06-25T00:58:26Z

Motivation

Cold-start cache initialization rebuilds each stripe's in-memory directory from disk on every
restart — multi-minute on large caches.

Approach

Host the directory (raw_dir) in POSIX shared memory so the next process start attaches the
existing segment in milliseconds instead of rebuilding it. Recovery is binary and fail-safe:
anything untrustworthy (crash, reboot, ABI/schema mismatch, failed validation, bad disk) falls
back to the existing disk-rebuild path, and reads still validate Doc magic + key so a stale
entry is a miss, never served corruption.

New configs & traffic_ctl commands

Opt-in behind proxy.config.cache.shm.enabled (default 0, a functional no-op). Also adds
proxy.config.cache.shm.{name_prefix,use_hugepages,purge_stale_on_start} and a
traffic_ctl cache shm status|clear command.

Details

Design, recovery model, configuration, and platform notes are in
doc/developer-guide/cache-architecture/shm-fast-restart.en.rst (added in this PR) and the
proxy.config.cache.shm.* entries in records.yaml.

Testing

Unit: test_CacheShm (ABI hash, storage signature, control round-trip, name length, prefix
normalization, process liveness).
AuTest: 7 cache_shm_* suites — fast restart, unclean shutdown, schema/storage mismatch,
bad-disk drop + orphan reclaim, concurrent-attach refusal, purge-on-disable.

Caveats

No automated test yet exercises the _shm_directory_is_valid() rejection branches or the
bad-disk / flush-failure invalidate_stripe_directory() paths (planned follow-up).
macOS concurrent-attach is best-effort: the flock guard is authoritative on Linux but a
no-op on macOS POSIX shm, where it falls back to an owner-pid liveness check.

Copilot

Pull request overview

This PR introduces an opt-in “fast restart” cache directory implementation by hosting each stripe’s in-memory directory (raw_dir) in POSIX shared memory, allowing subsequent traffic_server starts to attach the prior directory quickly instead of rebuilding from disk. It also adds operator tooling (traffic_ctl cache shm status|clear), configuration records, extensive AuTest coverage, unit tests for trust gates, and design/admin documentation.

Changes:

Add shared-memory directory infrastructure (CacheShm*) plus integration into cache startup, stripe directory allocation, and shutdown/clean-marking paths.
Add traffic_ctl cache shm status|clear commands and new proxy.config.cache.shm.* records.
Add unit + AuTest suites and documentation for the shm fast-restart feature.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/gold_tests/logging/ts_process_handler.py	Hardens psutil cmdline handling for macOS/permission-restricted processes.
tests/gold_tests/cache/shm_poke.py	Adds Linux-only helper to tamper with `/dev/shm` segments for trust-gate testing.
tests/gold_tests/cache/replay/cache-shm-fast-restart.replay.yaml	Proxy Verifier replay traffic used to validate hit-after-restart behavior.
tests/gold_tests/cache/gold/cache_shm_state_after_shutdown.gold	Gold output for `traffic_ctl cache shm status` validation.
tests/gold_tests/cache/cache_shm_unclean_shutdown.test.py	AuTest for rejecting dirty segments after SIGKILL and rebuilding from disk.
tests/gold_tests/cache/cache_shm_storage_mismatch.test.py	AuTest for partial attach behavior when storage layout changes.
tests/gold_tests/cache/cache_shm_schema_mismatch.test.py	Linux-only AuTest that pokes schema_version and verifies drop+rebuild.
tests/gold_tests/cache/cache_shm_purge_on_disable.test.py	AuTest for purge-on-disabled-start behavior and `traffic_ctl` exit codes.
tests/gold_tests/cache/cache_shm_fast_restart.test.py	End-to-end AuTest for clean shutdown -> shm attach -> cache HIT without origin contact.
tests/gold_tests/cache/cache_shm_concurrent_attach.test.py	AuTest for concurrent-attach guard (flock vs owner_pid liveness backstop).
tests/gold_tests/cache/cache_shm_bad_disk_dropped.test.py	AuTest for partial attach and orphan reclaim when a disk is removed from storage.
src/traffic_ctl/traffic_ctl.cc	Adds `traffic_ctl cache shm status
src/traffic_ctl/CMakeLists.txt	Builds the new traffic_ctl command source and adds cache include path for shared headers.
src/traffic_ctl/CacheShmCommand.h	Declares the `traffic_ctl cache shm` command handler.
src/traffic_ctl/CacheShmCommand.cc	Implements shm status/clear via direct `shm_open`/`mmap`, shared purge primitive, and exit codes.
src/records/RecordsConfig.cc	Registers `proxy.config.cache.shm.*` configuration records.
src/iocore/cache/unit_tests/test_CacheShm.cc	Adds unit tests for ABI hash, storage signature, layout round-trip, prefix normalization, and liveness checks.
src/iocore/cache/StripeSM.cc	Adds fast-attach path that can skip disk dir read + recovery when shm directory is trusted; adjusts shutdown behavior.
src/iocore/cache/Stripe.h	Adds `_shm_directory_is_valid()` and strengthens flush API via `[[nodiscard]]`.
src/iocore/cache/Stripe.cc	Uses shm-backed directory allocation; adds shm bounds validation; detaches shm mappings safely in destructor.
src/iocore/cache/CMakeLists.txt	Builds new `CacheShm.cc` and adds `test_CacheShm` to the cache unit tests.
src/iocore/cache/CacheShmPurge.h	Introduces shared header-only purge/enumerate/unlink primitive for server startup and traffic_ctl.
src/iocore/cache/CacheShmLayout.h	Defines the shared control-segment layout and prefix normalization utilities.
src/iocore/cache/CacheShm.h	Declares the CacheShm facade for lifecycle + stripe attach/create + trust gates.
src/iocore/cache/CacheShm.cc	Implements shm control segment lifecycle, trust gates, stripe attach/create, orphan reclaim, and clean-shutdown marking.
src/iocore/cache/CacheProcessor.cc	Calls CacheShm initialization before stripe construction and finalization after cache init.
src/iocore/cache/CacheDir.cc	Marks shm control segment clean only after shutdown sync has quiesced writers.
src/iocore/cache/AggregateWriteBuffer.h	Marks `flush()` as `[[nodiscard]]` to encourage handling short-write failures.
src/iocore/cache/AggregateWriteBuffer.cc	Converts flush failure from an assertion to a `false` return for graceful handling.
doc/developer-guide/cache-architecture/shm-fast-restart.en.rst	Adds detailed design doc: layout, gates, attach modes, shutdown semantics, tooling, and platform notes.
doc/developer-guide/cache-architecture/index.en.rst	Hooks shm fast-restart document into the cache architecture index.
doc/admin-guide/files/records.yaml.en.rst	Documents new `proxy.config.cache.shm.*` records and operator-facing behavior/tooling.

Copilot

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 4 comments.

bryancall

Approving. This is well built and safe to land. It is opt-in and off by default, the shared-memory attach path validates the mapping size before use and bounds the stripe count, and every failing system call is logged with its name and error string and then propagated back to the caller.

One correction to my earlier note. I wrote that a fast-attach bug returning a wrong offset or length would still pass CI. That is not accurate, and I withdraw it. The fast_restart test wires the origin to return a 502 on the hit transaction, so if the shared-memory directory failed to resolve the cached object, ATS would fall through to the origin and the 502 would fail the test. The real gap is narrower than I first stated.

The items below are all non-blocking for an off-by-default feature, but worth closing out before this is relied on in production.

The testing table in doc/developer-guide/cache-architecture/shm-fast-restart.en.rst lists a cache_shm_data_integrity autest that is not in this PR. Please either add the test or drop the row.
The objects the tests serve are small and single-fragment. A multi-fragment or large-object case after a fast attach would cover the one path the 502 trick does not fully exercise.
The rejection branches in _shm_directory_is_valid() and the invalidate-on-shutdown paths have no direct unit coverage, as the caveats note. They are close to pure header math, so each branch is a cheap unit test.

I also agree with the open Copilot threads on this PR.

In CacheShm.cc:229, unlink_all_known_segments() unlinks stripe names read from the control segment. On the path that drops an untrusted or corrupt control segment those names are untrusted, so a corrupt segment could lead to shm_unlink on unrelated objects. Since the names are always built from name_prefix via build_stripe_shm_name(), gating the unlink loop on a prefix match, and only running it when the control magic is valid, closes this with no effect on the normal path.
The three storage.config references in the design doc at lines 132, 340, and 430 should be storage.yaml to match the tests and the rest of this PR.

Thanks for the clean, well-documented change. The approval stands.

masaori335 · 2026-07-08T05:48:32Z

Thank you for taking a look. I addressed 1. 4. and 5. by 0cc3b52.

Copilot

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/iocore/cache/AggregateWriteBuffer.cc:52

AggregateWriteBuffer::flush() relies on a single pwrite() writing the entire buffer and stores the return value in an int. On regular files, pwrite() can legally return a short write or fail with EINTR, and the narrowing cast can mis-handle larger writes. With the new shutdown path treating a flush failure as a reason to invalidate the shm directory, these false negatives can unnecessarily drop the fast-restart optimization (and potentially lose buffered data on shutdown when the error was transient).

AggregateWriteBuffer::flush(int fd, off_t write_pos) const
{
  int r = pwrite(fd, this->_buffer, this->_buffer_pos, write_pos);
  if (r != this->_buffer_pos) {
    return false;

src/iocore/cache/Stripe.cc:224

Stripe::_shm_directory_is_valid() does multiple full passes over every directory entry on fast-restart attach (a full segment_entries scan, plus _shm_segment_membership_is_valid()'s visited.assign(...) + final for scan, plus directory.check_segment(s) walking all buckets). For large caches this is O(directory_size) in CPU and can substantially eat into the “fast restart” benefit even when the segment is clean and valid.

    for (int64_t i = 0; i < segment_entries; i++) {
      Dir *e = dir_in_seg(seg, i);
      if (dir_next(e) >= segment_entries) {
        return false;
      }

Cold-start cache initialization rebuilds each stripe's in-memory directory from disk on every restart, which is multi-minute on large caches. Host the directory in POSIX shared memory so the next process start attaches the existing segment in milliseconds instead of rebuilding it. Recovery stays binary and fail-safe: when the segment cannot be trusted -- crash, reboot, ABI or schema mismatch, storage change, or failed validation -- the start drops it and rebuilds through the existing disk path. Reads still validate Doc magic and key, so a stale entry is a miss and never corruption. A stripe the previous shutdown could not vouch for is marked in the control segment, never in the stripe's own header: that header aliases raw_dir, which is also the source buffer for the on-disk directory write, so a mark there could reach disk and make the next start clear the stripe instead of recovering it. Opt-in behind proxy.config.cache.shm.enabled, default 0, where it is a functional no-op. `traffic_ctl cache shm status` and `clear` inspect and drop segments out of band. The design and the full recovery matrix are in doc/developer-guide/cache-architecture/shm-fast-restart.en.rst.

The status/clear leaves passed their own `[&]() { command->execute(); }` lambda, where every other leaf in the file passes Command_Execute. Reuse it so the null guard applies and the wiring is uniform. No behavior change: `command` is assigned before args.invoke() and a null one throws, so the guard cannot fire today.

The untrusted-entry test simulated a restart by calling initialize() again in one process, which the concurrent-attach guard correctly refuses: the first start still holds LOCK_EX on the control fd, and the second open of the same object conflicts with it. The test only passed where flock is not honored for POSIX shm (macOS, FreeBSD) and failed on Fedora and Debian; the other Linux builds compile the feature out, so the target is not built there at all. Add CacheShm::release_for_test() to drop the process-wide state, standing in for the process exit that releases the flock in production.

Skipping the write for a shm-backed stripe looked free -- the segment is already current and is attached directly next start -- but the on-disk copy is the only thing the fallback has, and recover_data() cannot always rebuild from it. handle_recover_from_data() returns without scanning the data region when the on-disk header still has sync_serial == 0, so an empty directory is accepted as-is. A stripe filled and cleanly shut down before the first periodic dir sync (60 s by default) is exactly that case: nothing had written the on-disk dir, so the next start that cannot use the segment found an empty directory and lost every object. cache_shm_dir_invalid caught it as a 502 against its deliberately absent origin, once the poked segment was correctly rejected. Only start time is what this feature set out to improve, so the shutdown write costs what it did before and the fallback stays recoverable.

Review of the fast-restart path found four ways a shared-memory segment could be trusted, or cleared, when it should not be. The attach gate bounded directory links but neither a live entry's offset -- which CacheVC::handleRead turns into a negative, so huge unsigned, read length -- nor the free list's structure, where an in-range cycle lets freelist_pop write a link over a live entry's tag bits. Clean shutdown cleared owner_pid while event threads were still writing, which on a platform where flock is a no-op is the only guard against a concurrent attach. And traffic_ctl swept a control segment smaller than this build's before checking for a live owner, so a newer build could unlink a running older build's segments -- the upgrade case the frozen header exists to support. The entry bound is on where an entry starts, not on its extent: dir_approx_size rounds up, so the last object in a stripe legitimately overhangs the stripe end, which is what handleRead's truncation is for.

A walk from the free-list head cannot see an entry that Directory::insert unlinked but never filled, so a shutdown torn in that window published a directory whose stale prev/next the next insert would write through. Require every entry to be reached exactly once, as a bucket root, an empty free-list node, or an in-use chain node, and delay the clean-shutdown mark until after the event system is down so fewer tears reach the gate at all.

Copilot

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/traffic_ctl/CacheShmCommand.cc:65

shm_segment_exists() reports a stripe segment as absent whenever shm_open() fails. That includes EACCES, which means the segment likely exists but is not readable by the current user; printing MISSING in that case is misleading during troubleshooting.

Treat EACCES as “exists” (or surface the errno in the output) so traffic_ctl cache shm status doesn’t incorrectly claim a segment is missing.

bool
shm_segment_exists(const std::string &name)
{
  int fd = shm_open(name.c_str(), O_RDONLY, 0);
  if (fd < 0) {
    return false;
  }
  close(fd);
  return true;
}

Pointed out issues are addressed.

* cache: shared-memory-backed Dir for fast restart Cold-start cache initialization rebuilds each stripe's in-memory directory from disk on every restart, which is multi-minute on large caches. Host the directory in POSIX shared memory so the next process start attaches the existing segment in milliseconds instead of rebuilding it. Recovery stays binary and fail-safe: when the segment cannot be trusted -- crash, reboot, ABI or schema mismatch, storage change, or failed validation -- the start drops it and rebuilds through the existing disk path. Reads still validate Doc magic and key, so a stale entry is a miss and never corruption. A stripe the previous shutdown could not vouch for is marked in the control segment, never in the stripe's own header: that header aliases raw_dir, which is also the source buffer for the on-disk directory write, so a mark there could reach disk and make the next start clear the stripe instead of recovering it. Opt-in behind proxy.config.cache.shm.enabled, default 0, where it is a functional no-op. `traffic_ctl cache shm status` and `clear` inspect and drop segments out of band. The design and the full recovery matrix are in doc/developer-guide/cache-architecture/shm-fast-restart.en.rst. * traffic_ctl: run cache shm subcommands through Command_Execute The status/clear leaves passed their own `[&]() { command->execute(); }` lambda, where every other leaf in the file passes Command_Execute. Reuse it so the null guard applies and the wiring is uniform. No behavior change: `command` is assigned before args.invoke() and a null one throws, so the guard cannot fire today. * cache: fix shm shutdown test where flock is honored The untrusted-entry test simulated a restart by calling initialize() again in one process, which the concurrent-attach guard correctly refuses: the first start still holds LOCK_EX on the control fd, and the second open of the same object conflicts with it. The test only passed where flock is not honored for POSIX shm (macOS, FreeBSD) and failed on Fedora and Debian; the other Linux builds compile the feature out, so the target is not built there at all. Add CacheShm::release_for_test() to drop the process-wide state, standing in for the process exit that releases the flock in production. * cache: always write the on-disk dir at shm shutdown Skipping the write for a shm-backed stripe looked free -- the segment is already current and is attached directly next start -- but the on-disk copy is the only thing the fallback has, and recover_data() cannot always rebuild from it. handle_recover_from_data() returns without scanning the data region when the on-disk header still has sync_serial == 0, so an empty directory is accepted as-is. A stripe filled and cleanly shut down before the first periodic dir sync (60 s by default) is exactly that case: nothing had written the on-disk dir, so the next start that cannot use the segment found an empty directory and lost every object. cache_shm_dir_invalid caught it as a 502 against its deliberately absent origin, once the poked segment was correctly rejected. Only start time is what this feature set out to improve, so the shutdown write costs what it did before and the fallback stays recoverable. * cache: harden the shm trust gates and ownership guard Review of the fast-restart path found four ways a shared-memory segment could be trusted, or cleared, when it should not be. The attach gate bounded directory links but neither a live entry's offset -- which CacheVC::handleRead turns into a negative, so huge unsigned, read length -- nor the free list's structure, where an in-range cycle lets freelist_pop write a link over a live entry's tag bits. Clean shutdown cleared owner_pid while event threads were still writing, which on a platform where flock is a no-op is the only guard against a concurrent attach. And traffic_ctl swept a control segment smaller than this build's before checking for a live owner, so a newer build could unlink a running older build's segments -- the upgrade case the frozen header exists to support. The entry bound is on where an entry starts, not on its extent: dir_approx_size rounds up, so the last object in a stripe legitimately overhangs the stripe end, which is what handleRead's truncation is for. * cache: prove shm segment membership at attach A walk from the free-list head cannot see an entry that Directory::insert unlinked but never filled, so a shutdown torn in that window published a directory whose stale prev/next the next insert would write through. Require every entry to be reached exactly once, as a bucket root, an empty free-list node, or an in-use chain node, and delay the clean-shutdown mark until after the event system is down so fewer tears reach the gate at all. * Include algorithm header (cherry picked from commit fc115b1)

cmcfarlen · 2026-08-10T15:27:53Z

Cherry-picked to the 10.2.x branch as abfbfcd for the 10.2.0 release.

Picks up apache#13328 (shared-memory cache directory for fast restart) and apache#13418 (traffic_ctl cache clear). The shm directory gets its own section since it is a new opt-in feature with four new records and a traffic_ctl subcommand.

* Add 10.2.0 changelog and release notes Generate CHANGELOG-10.2.0 from the 10.2.0 milestone and document the release in whats-new and upgrading. The connect retry change (#13102) is called out as a necessary incompatible change, since the retry limits were not previously applied according to origin state. * Address review: fix PR count and token_key markup The PR count was 655 before five stale milestone entries were dropped; the changelog has 650. Use :ts:cv: for proxy.config.quic.server.token_key.filename, which is documented on 10.2.x even though it is absent from master, where it was first checked. * Add late 10.2.x additions to changelog and release notes Picks up #13328 (shared-memory cache directory for fast restart) and #13418 (traffic_ctl cache clear). The shm directory gets its own section since it is a new opt-in feature with four new records and a traffic_ctl subcommand. * Add July 2026 security fixes to changelog and release notes The Release 2 security bundle (#13452) landed directly on 10.2.x without public PRs, so those commits never appear in a milestone. Source them from the commit range with the changelog tool's git-range mode and append them as bare subjects, matching how CHANGELOG-10.1.4 lists them. Link the advisory from whats-new for the CVE mapping. * Add #13352, #13517 and #13523 to the changelog Three late bug fixes on 10.2.x. All are fixes with no new configuration, metrics or API surface, so only the changelog and the commit/PR counts change.

masaori335 added this to the 11.0.0 milestone Jun 25, 2026

masaori335 self-assigned this Jun 25, 2026

Copilot AI lite review requested due to automatic review settings June 25, 2026 00:58

masaori335 added the Cache label Jun 25, 2026

masaori335 added this to ATS v10.2.x Jun 25, 2026

Copilot started reviewing on behalf of masaori335 June 25, 2026 00:58 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread src/iocore/cache/CacheShmPurge.h Outdated

masaori335 force-pushed the asf-master-dir-on-shm branch from 8358d37 to e0ee0af Compare June 25, 2026 03:34

masaori335 requested a review from Copilot June 25, 2026 03:35

Copilot started reviewing on behalf of masaori335 June 25, 2026 03:35 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Comment thread src/iocore/cache/CacheShmPurge.h

Comment thread src/iocore/cache/CacheShmLayout.h Outdated

bryancall requested review from bryancall and moonchen June 29, 2026 22:32

moonchen requested a review from Copilot June 29, 2026 22:32

Copilot started reviewing on behalf of moonchen June 29, 2026 22:33 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

bryancall previously approved these changes Jul 6, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 8, 2026 05:47

masaori335 dismissed bryancall’s stale review via 0cc3b52 July 8, 2026 05:47

Copilot started reviewing on behalf of masaori335 July 8, 2026 05:47 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Comment thread src/iocore/cache/CacheShm.cc

Comment thread src/iocore/cache/CacheShm.cc Outdated

moonchen requested changes Jul 8, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 14, 2026 04:10

Copilot started reviewing on behalf of masaori335 July 14, 2026 04:10 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread src/iocore/cache/CacheShmPurge.h Outdated

Comment thread doc/admin-guide/files/records.yaml.en.rst

Copilot AI review requested due to automatic review settings July 14, 2026 04:37

masaori335 force-pushed the asf-master-dir-on-shm branch from 91899f8 to 161f39c Compare July 14, 2026 04:37

Copilot started reviewing on behalf of masaori335 July 14, 2026 04:37 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread src/iocore/cache/CacheShm.cc

Comment thread tests/gold_tests/logging/ts_process_handler.py Outdated

Copilot started reviewing on behalf of masaori335 August 7, 2026 00:58 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 7, 2026 01:02

Copilot started reviewing on behalf of masaori335 August 7, 2026 01:03 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

masaori335 added 7 commits August 7, 2026 10:16

Include algorithm header

cd4bff9

Copilot AI review requested due to automatic review settings August 7, 2026 01:22

masaori335 force-pushed the asf-master-dir-on-shm branch from 2cfbb88 to cd4bff9 Compare August 7, 2026 01:22

Copilot started reviewing on behalf of masaori335 August 7, 2026 01:23 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

bneradt approved these changes Aug 7, 2026

View reviewed changes

masaori335 merged commit fc115b1 into apache:master Aug 7, 2026
15 checks passed

github-project-automation Bot moved this to For v10.2.0 in ATS v10.2.x Aug 7, 2026

This was referenced Aug 9, 2026

Cherry-picks for 10.2.0 RC, round 3 (2026-08-09) #13520

Merged

Cherry-picks for 10.2.0 RC, round 4: cache shm fast restart + prerequisite (2026-08-10) #13522

Merged

cmcfarlen moved this from For v10.2.0 to Picked v10.2.0 in ATS v10.2.x Aug 10, 2026

cmcfarlen modified the milestones: 11.0.0, 10.2.0 Aug 10, 2026

Uh oh!

Conversation

masaori335 commented Jun 25, 2026

Motivation

Approach

New configs & traffic_ctl commands

Details

Testing

Caveats

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bryancall left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

masaori335 commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

cmcfarlen commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

bryancall left a comment •

edited

Loading