Repository navigation
Add a fetch governor: dedupe, negative-cache and back off repeated proxy fetches - #79
Merged
Merged
Conversation
All proxied requests now pass through a governor that coalesces concurrent fetches of the same URL, negative-caches failures in Dexie, backs off transient errors with exponential jitter, and opens a circuit breaker on HTTP 429 so a capped CORS proxy (Cloudflare 1027) cannot be hammered. Images that fail during ingest are replaced with a local placeholder so a saved article never keeps a remote URL that every read would re-request. Refs #75
✅ Deploy Preview for savrlist ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
jonocodes
added a commit
that referenced
this pull request
Oct 4, 2026
Conflicts: - ArticleComponent.tsx: #80 moved sanitising into sanitizeArticleHtml (video-embed iframe allow-list) but still passed a fresh `{ __html }` literal each render, so the DOM-rebuild bug remained on main. Keep #80's sanitiser and this branch's memoization: useMemo(() => ({ __html: sanitizeArticleHtml(html) }), [html]). - CHANGELOG.md: keep both; this branch's entries move under #80's 2026-10-03 heading since they land after it. With embeds now rendered the rebuild also reloaded every player on each save: on main's code 3 embeds were loaded 18 times over 3 reading pauses. Added a scroll-stability test pinning one load per embed (fails on main's version of the line, passes with the memoization). Adjusted for #79, which swaps failed images for a local placeholder: - The failed-download dimensions test now expects the placeholder and data-orig-src, and still asserts no width/height is claimed. - The shared jest mock of ~/utils/article/tools gains a FetchError class. ingestion.ts now does `e instanceof FetchError` on a failed image, and without it the check throws; no existing test ran a failed download through ingestHtml, so the gap was latent. - Comment and changelog wording: only articles saved before #79 still hold remote URLs for failed images. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a fetch governor in front of every proxied request. All article and image fetches already funnel through
fetchWithTimeout(src/utils/article/tools.ts), so the governor lives there and covers everything:permanent4xx /transienttimeout, 5xx, 429) and stored in a new Dexie table (fetchCache, db v6) so a known-bad URL is not retried until itsnextAttemptAtpasses.1027, free-tier cap) opens a breaker and short-circuits all proxied fetches for a cooldown.clearFetchFailures()so a user-forced retry isn't blocked by a backoff window.It also closes the reader half of #75: an image that fails during ingest is replaced with a local placeholder (
applyImagePlaceholder), with the original kept indata-orig-src, so a saved article never re-requests a dead remote URL.Why
On 2026-10-02 the shared CORS worker served ~234k invocations against a 10–50/day baseline and hit the Workers Free daily cap. ~10 image URLs returning 404 were each fetched ~24k times over 14h: the upstream 404 sends
cache-control: no-cache, and the app had no de-dupe, no negative cache, and no backoff. Ticket: #78.Stacked PR
Based on
feature/proxy-failure-context(#77) because it usesFetchError.requestUrl/status. Retarget tomainonce #77 merges.Testing
src/utils/net/fetchGovernor.test.ts(classification, coalescing, permanent/transient, backoff, breaker, clear/clearAll) andlib/__tests__/ingestion-placeholder.test.ts.tests/e2e/ingest-failed-image.spec.tsagainst a new fixture with one good and one 404 image, asserting the placeholder + no remotesrc.npx jest→ 327 passed;tsc --noEmit,eslint, andvite buildclean.flox activate -c "node scripts/run-e2e.js tests/e2e/smoke.spec.ts tests/e2e/ingest-failed-image.spec.ts"→ 4 passed.docs/DEVELOPMENT.mddocuments this.Closes #78. Refs #75.