feat: Add X-IPFS-Root-CID header for faster cache invalidation

[mathew-cf]

If content resolution is performed separately from a go-ipfs node via DNS and/or IPNS, then the root CID can be used to check if a cached response is fresh or stale for a given path.

This allows quick cache invalidation once a change to a DNSLink record is propagated.

[mathew-cf]

ipfs-webui test is flaky. amending and pushing

[mathew-cf]

tests still flaky. amending and pushing again

[Stebalien]

Note: If you're just caching, use the etag. It's the CID when downloading a file, and the CID plus a dir-index hash when indexing a directory.

(otherwise, it does seem reasonable to say "here's the CID of the thing I just resolved, but I'd just call it x-ipfs-cid in that case).

[mathew-cf]

Basically the idea here is that I can use (mostly) DNS to resolve to a root CID at the edge. I can quickly cache and serve new data if I know whether my data currently in cache corresponds to the DNS record.

The resolved root CID is different from the CIDs of the files in the directory (i.e. the CID in the etag). I named the header x-ipfs-root-cid rather than x-ipfs-cid to try to make that clearer.

[mathew-cf]

This wouldn't support IPFS subpaths. Needs some more thought

[lidel]

I understand the desire here is to have resolved IPNS root CID in the response?
Sounds good, but agree this needs more thought.

• With the main Root CID + Etag you could do efficient invalidation of entire website every time IPNS mapping changes.
  • I think if we return all intermediate roots as well, that will enable more fine-grained cache invalidation at the path level – FYSA are looking into landing cache improvements like this – feel free to pitch in ideas/requests in Meta: HTTP Gateway cache control improvements #8717 🙏

[lidel]

@mathew-cf check #8720