feat: added purger_http module - #39
Conversation
…io modules. (#40) * feat: added purger_http module and retire the section.io related module and settings. * Fix CloudFront cache tag invalidation tokens Prefix bare tag invalidations with tag: so the invalidation API treats them as CloudFront cache tags. Support comma, pipe and tab separators, preserve existing markers and deduplicate expressions. * Empty the purge queue when switching to the Marina CloudFront purger Items queued for the Section.io purger (URL and wildcard invalidations) can never be handled by the tag-only Marina purger. Purge marks them NOT_SUPPORTED and returns them to the queue, so they recirculate on every run and, because the queue is ordered by creation time, always occupy the first slot of every claimed batch. purge_tokens_tokens() only builds token replacements when the batch offered to a purger starts at index 0, so one stale item is enough to send every real invalidation with the literal [invalidations:separated_comma] token as its request body. bay_platform_dependencies_update_10007() empties the queue once so environments migrated from Section.io start clean. * Hash cache tags for CloudFront and fix the tags header format CloudFront's tag invalidation indexes the x-amz-meta-cache-tag response header as a comma-separated list, rejects tags containing spaces, keeps at most 50 tags per object and caps the header at 1,783 characters. The header this module emitted was space-separated and unbounded, so a "#tag" invalidation could never match anything. The header now filters tags against the purge_queuer_coretags blacklist, orders entity tags first, hashes each tag with xxHash3 truncated to six hex characters, de-duplicates and caps the list at 50, and joins it with commas. The invalidation side uses the same hash service and prefixes each value with "tag:", which the cache invalidation API rewrites to "#<hash>". Both ends therefore always agree. The token replacement is also rebuilt unconditionally in hook_tokens_alter(): purge_tokens_tokens() checks $data['invalidations'][0] and silently produces nothing when the batch offered by PurgersService does not start at index 0, which happens whenever an unsupported item sits ahead in the queue. The hash, filter and prioritizer are services that sites can override. Adapted from the cloudfront_purger_tags submodule of drupal/cloudfront_purger (GPL-2.0-or-later). * Patch Purge invalidation token replacement Apply the patch for Drupal issue #3484260 to prevent token replacement from being skipped when invalidation sets lack index 0.
| public function filter(array $tags): array { | ||
| // cspell:ignore-word blacklist -- Purge's config key. | ||
| $blocklist = $this->configFactory | ||
| ->get('purge_queuer_coretags.settings') |
There was a problem hiding this comment.
Are we using this yet? Any examples of entities where updating would not need an invalidation?
There was a problem hiding this comment.
hey @nicksantamaria, Thanks for the review.
It is not about skipping invalidations. the purge_queuer_coretags blacklist already stops those tags being queued, so they're never invalidated regardless. The filter is only about the response header: it keeps x-amz-meta-cache-tag aligned with what can actually be invalidated, so blacklisted tags don't consume slots against CloudFront's 50-tags-per-object limit (a Drupal page easily emits 30+ tags). It's a service so a site can override it, and it's a no-op when the blacklist is empty. Happy to drop it if you'd rather keep the module lean - it's an optimisation, not required for correctness.
There was a problem hiding this comment.
OK thanks for the clarification - so this is glue code to ensure the response header only includes cache tag types that would trigger invalidations.
I think this is a good thing to implement - off the top of my head media and paragraph entities should not create invalidations so excluding them would be a great optimisation.
| * - `tag:<x>` becomes `#<x>` (a CloudFront cache tag) | ||
| * - starts with `/`, `#`, or contains `:`: passes through unchanged | ||
| * - anything else gets a leading `/` (a CloudFront path) |
There was a problem hiding this comment.
The invalidation API actually handles these transforms automatically - just need the raw tags (although it will need the hashed version).
There was a problem hiding this comment.
hey @nicksantamaria ,
Good to know that's the intent, but I don't think the currently deployed API does it yet. In publisher.ts the VTL only produces a #-prefixed cache tag from the tag: branch; anything else with no /, # or : gets a leading /, so a bare hash like b46795 normalises to /b46795, and a raw node:203 stays node:203. The tag:<hash> form is the one branch that yields #<hash>, and it's what actually purged end to end on PR-675 (verified: edit → age reset → new title for anonymous users)
If you're planning to have the API normalise raw tags itself, happy to drop the prefix once that's in. Which side should own the # conversion?
There was a problem hiding this comment.
Hey if you look at the VTL script here, it converts tag: to # https://github.com/dpc-sdp/marina/blob/main/src/infra/constructs/cache-invalidation/publisher.ts#L149
I've also tested it just now and can confirm its working as intended. Ditch the transforms here :)
There was a problem hiding this comment.
hey @nicksantamaria ,
the change has been applied, the transform is removed, it now sends the bare hashed tags!
The cache invalidation API's VTL normalises the tag itself, so the module no longer prefixes invalidations with `tag:`. Each tag invalidation is sent as the bare xxHash3 hash — the same value emitted in the x-amz-meta-cache-tag response header — and the API converts it to a CloudFront cache tag.
3 participants
the changes are also based on 2 resources listed below