Cherry-picks for 10.2.0 RC, round 4: cache shm fast restart + prerequisite (2026-08-10) - #13522
Merged
Merged
Conversation
Cache resets currently require restarting Traffic Server or manually changing the cache generation. This complicates repeatable workload comparisons and their automation. This adds a traffic_ctl cache clear command backed by a restricted JSON-RPC method that advances the global HTTP cache generation. This also documents the logical purge semantics and exercises the command in the cache-generation AuTest. Fixes: apache#9399 (cherry picked from commit 2d9d666)
* cache: shared-memory-backed Dir for fast restart
Cold-start cache initialization rebuilds each stripe's in-memory
directory from disk on every restart, which is multi-minute on large
caches. Host the directory in POSIX shared memory so the next process
start attaches the existing segment in milliseconds instead of
rebuilding it.
Recovery stays binary and fail-safe: when the segment cannot be
trusted -- crash, reboot, ABI or schema mismatch, storage change, or
failed validation -- the start drops it and rebuilds through the
existing disk path. Reads still validate Doc magic and key, so a
stale entry is a miss and never corruption. A stripe the previous
shutdown could not vouch for is marked in the control segment, never
in the stripe's own header: that header aliases raw_dir, which is
also the source buffer for the on-disk directory write, so a mark
there could reach disk and make the next start clear the stripe
instead of recovering it.
Opt-in behind proxy.config.cache.shm.enabled, default 0, where it is
a functional no-op. `traffic_ctl cache shm status` and `clear`
inspect and drop segments out of band. The design and the full
recovery matrix are in
doc/developer-guide/cache-architecture/shm-fast-restart.en.rst.
* traffic_ctl: run cache shm subcommands through Command_Execute
The status/clear leaves passed their own `[&]() { command->execute(); }`
lambda, where every other leaf in the file passes Command_Execute.
Reuse it so the null guard applies and the wiring is uniform. No
behavior change: `command` is assigned before args.invoke() and a null
one throws, so the guard cannot fire today.
* cache: fix shm shutdown test where flock is honored
The untrusted-entry test simulated a restart by calling initialize()
again in one process, which the concurrent-attach guard correctly
refuses: the first start still holds LOCK_EX on the control fd, and the
second open of the same object conflicts with it. The test only passed
where flock is not honored for POSIX shm (macOS, FreeBSD) and failed on
Fedora and Debian; the other Linux builds compile the feature out, so
the target is not built there at all.
Add CacheShm::release_for_test() to drop the process-wide state,
standing in for the process exit that releases the flock in production.
* cache: always write the on-disk dir at shm shutdown
Skipping the write for a shm-backed stripe looked free -- the segment is
already current and is attached directly next start -- but the on-disk
copy is the only thing the fallback has, and recover_data() cannot
always rebuild from it. handle_recover_from_data() returns without
scanning the data region when the on-disk header still has
sync_serial == 0, so an empty directory is accepted as-is.
A stripe filled and cleanly shut down before the first periodic dir
sync (60 s by default) is exactly that case: nothing had written the
on-disk dir, so the next start that cannot use the segment found an
empty directory and lost every object. cache_shm_dir_invalid caught it
as a 502 against its deliberately absent origin, once the poked segment
was correctly rejected.
Only start time is what this feature set out to improve, so the
shutdown write costs what it did before and the fallback stays
recoverable.
* cache: harden the shm trust gates and ownership guard
Review of the fast-restart path found four ways a shared-memory segment
could be trusted, or cleared, when it should not be. The attach gate
bounded directory links but neither a live entry's offset -- which
CacheVC::handleRead turns into a negative, so huge unsigned, read length
-- nor the free list's structure, where an in-range cycle lets
freelist_pop write a link over a live entry's tag bits. Clean shutdown
cleared owner_pid while event threads were still writing, which on a
platform where flock is a no-op is the only guard against a concurrent
attach. And traffic_ctl swept a control segment smaller than this build's
before checking for a live owner, so a newer build could unlink a running
older build's segments -- the upgrade case the frozen header exists to
support.
The entry bound is on where an entry starts, not on its extent:
dir_approx_size rounds up, so the last object in a stripe legitimately
overhangs the stripe end, which is what handleRead's truncation is for.
* cache: prove shm segment membership at attach
A walk from the free-list head cannot see an entry that
Directory::insert unlinked but never filled, so a shutdown torn in
that window published a directory whose stale prev/next the next
insert would write through. Require every entry to be reached exactly
once, as a bucket root, an empty free-list node, or an in-use chain
node, and delay the clean-shutdown mark until after the event system
is down so fewer tears reach the gate at all.
* Include algorithm header
(cherry picked from commit fc115b1)
cmcfarlen
force-pushed
the
10.2.x-picks-20260810
branch
from
August 10, 2026 11:15
66d1040 to
abfbfcd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth round of cherry-picks for the 10.2.0 release candidate: #13328 and its prerequisite #13418, both at "For v10.2.0" in the ATS v10.2.x project.
#13328 was deferred from round 3 because it could not be picked on its own. #13418 is the prerequisite: it introduces the
traffic_ctl cachecommand group and theCacheCommandclass that #13328'scache shm status/cache shm clearsubcommands attach to. Picking #13328 alone lefttraffic_ctl.ccreferencing an undeclaredcache_commandand an unknownCacheCommand, which the compiler caught but the merge did not.Both picked with
git cherry-pick -xin master merge order. Three adaptations were needed; everything else applied unchanged.1.
src/traffic_ctl/traffic_ctl.cc(both picks) — dropped master-only command registrations. Auto-merge tried to bring across theconfig ssl-multicertandconfig convertsubcommand trees and an#include "ConvertConfigCommand.h", none of which belong to either PR; they come from master's YAML config conversion work and their implementation files do not exist on this branch. Only each PR's own additions were taken. The resulting per-PR diff for this file is byte-identical to master's for both picks.2.
src/traffic_ctl/CMakeLists.txt(#13328) — same drag-in. Auto-merge wantedConvertConfigCommand.ccandSSLMultiCertCommand.ccalongsideCacheShmCommand.cc; only the latter was taken.cmake-formatthen expanded the source list to one entry per line because adding it pushed the line past the width limit, so this file shows a larger diff than master's+1. The net content is the same:CacheShmCommand.ccadded, no master-only entries.3. The nine
cache_shm_*autests — convertedstorage.yamltostorage.config+volume.config. This branch's server readsstorage.configandvolume.config(ts::filename::STORAGEis"storage.config"here versus"storage.yaml"on master), and autest exposesDisk.storage_config/Disk.volume_configaccordingly, soDisk.storage_yamldoes not exist here. Each test's single span plus single volume becomes:cache_shm_bad_disk_droppedbuilds N spans in a loop and was converted to emit onestorage.configline per span.Dropping the per-span
name:field is safe for the shm mechanism: segments are keyed bystripe_key_hash, andinclude/shared/cache_shm/Layout.hstates this explicitly — "segment on attach by stripe_key_hash, not by name (order-independent)". The remainingstorage.yamlmentions in the picked files are all prose (docstrings, comments, and the developer guide); no test waits on a diags string or gold file containing it.Both adaptations were amended into their own picks rather than added as follow-up commits, so this branch is exactly two commits.
Verified locally: build clean, and
ctestis 167/167 — including the two unit tests #13328 adds,test_cache_CacheShmandtest_cache_CacheShmShutdown, which exercise the new mechanism rather than merely compiling it. All eightproxy.config.*records the new autests reference resolve inRecordsConfig.cc, and no picked test uses aDisk.*attribute orTest.*helper missing from this branch.4. Documentation and prose adapted to this branch's filenames. The picked docs described the feature in terms of
storage.yaml, which does not exist in 10.2.0. Converted tostorage.configindoc/admin-guide/files/records.yaml.en.rst(one:file:role) and the newdoc/developer-guide/cache-architecture/shm-fast-restart.en.rst(three literals, including the "storage change" row of the invalidation table), plus thestorage_signaturecomment ininclude/shared/cache_shm/Layout.hand seven docstring/comment references across thecache_shm_*autests. These are literal substitutions only: no headings, markup, or:ref:targets were touched, and all three:ref:targets the new guide uses (admin-cache-shm-fast-restart,cache-directory,shm-marking-untrusted) resolve on this branch. There is now no remaining mention ofstorage.yamlanywhere in the picked change.Draft so the full CI matrix runs before the release branch moves; the nine adapted autests are the main thing to watch. It will be landed by fast-forward.