Skip to content

fix(node): gate GET /ipfs/{cid} tree objects so a withheld subtree's structure can't leak (#135)#173

Open
beardthelion wants to merge 11 commits into
mainfrom
fix/issue-135-ipfs-cid-tree-gate
Open

fix(node): gate GET /ipfs/{cid} tree objects so a withheld subtree's structure can't leak (#135)#173
beardthelion wants to merge 11 commits into
mainfrom
fix/issue-135-ipfs-cid-tree-gate

Conversation

@beardthelion

@beardthelion beardthelion commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

What

GET /ipfs/{cid} served tree objects of a withheld subtree to callers denied that subtree. A git tree body is <mode> <name>\0<raw-oid> per entry, so fetching the tree CID of a withheld directory returned every child filename and child oid in cleartext, recursively. Blob content was already protected; this closes the structure leak so the CID surface matches what get_tree enforces on the REST path.

Approach

Tree objects are now gated against a caller-aware allowed-tree-set, the mirror of the existing allowed_blob_set_for_caller:

  • A shared object_paths walk (git ls-tree -rzt per reachable commit) that blob_paths and the new tree_paths both filter, so the two gates cannot drift. blob_paths output is byte-identical, so its callers are unaffected.
  • tree_paths is the kind == "tree" slice plus each reachable commit's root tree (resolved in one git log --format=%T pass, since ls-tree never emits a commit's own root).
  • get_by_cid gates a blob against the allowed-blob-set and a tree against the allowed-tree-set (lazy per repo, off the async runtime, fail-closed on any walk error). Commits and tags stay served: they expose only root-level metadata the caller already cleared the / gate for.

The withheld directory's own tree (path /secret) is denied, not just its descendants, so parity with get_tree holds.

Reachability and scope

get_by_cid resolves a CID by treating its sha256 digest as a git oid, which only matches in sha256 repos; production repos currently init --object-format=sha1, so the endpoint is largely dormant until a sha256 migration. This lands the gate ahead of that. The replication/pin path exports the same withheld-tree structure to IPFS independently of the object format and is the more urgent sibling, tracked separately in #172.

Tests

Deny paths driven through the real handler:

  • Withheld subtree tree CID returns 404 for anon and non-readers; listed reader and owner still read it (200).
  • Root and ancestor trees, commits, and tags stay served; a dangling tree fails closed for anon and owner; a tree with no path-scoped rule is served.
  • The withheld directory's own path denies (glob parity with get_tree).
  • Content-dedup: a tree reachable at both an allowed and a withheld path is served (allowed wins).
  • blob_paths output is asserted byte-identical after the walk refactor.

Closes #135.

Summary by CodeRabbit

  • New Features

    • Added improved per-path IPFS visibility controls that gate directory trees and file blobs.
    • Enhanced /ipfs/{cid} to resolve via pinned CID → git-object OID mapping with visibility-aware tree handling.
    • Introduced a per-client rate limiter for IPFS full-history walk requests.
  • Bug Fixes

    • Strengthened fail-closed behavior so unverifiable access and dangling objects are withheld (404).
    • Hardened object enumeration/parsing to prevent visibility leaks from malformed/unreachable data.
  • Tests

    • Expanded CID gating coverage with raw-byte tree verification, updated fixtures, and added denied-directory and dangling-tree fail-closed tests.
  • Chores

    • Added a database index and CID→OID lookup helper to speed up resolution.

@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Path-scoped IPFS visibility now applies to blob and tree objects using caller-specific reachable allow-sets. CID resolution uses pinned_cids, traversal fails closed, full-history walks are rate-limited, and tests cover withheld, allowed, root, shared, and dangling objects.

Changes

Path-scoped object visibility

Layer / File(s) Summary
Pinned CID resolution
crates/gitlawb-node/src/db/mod.rs, crates/gitlawb-node/src/api/ipfs.rs
CID pins are indexed and resolved to Git OIDs through pinned_cids before repository scanning.
Reachability and allow-set computation
crates/gitlawb-node/src/git/visibility_pack.rs, crates/gitlawb-node/src/visibility.rs
Shared fail-closed traversal enumerates reachable commits, blob paths, tree paths, and root trees for caller-specific allow sets, including subtree own-path rules.
IPFS blob and tree gating
crates/gitlawb-node/src/api/ipfs.rs
The handler memoizes separate blob and tree allow sets and skips repositories when computation fails or panics.
Rate limiting and regression coverage
crates/gitlawb-node/src/main.rs, crates/gitlawb-node/src/state.rs, crates/gitlawb-node/src/test_support.rs, crates/gitlawb-node/src/git/visibility_pack.rs
IPFS full-history walks use per-source limits, while tests cover pinned CIDs, withheld and dangling trees, raw tree contents, metadata objects, and rate-limit behavior.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant IPFSHandler
  participant Database
  participant VisibilityPack
  participant GitRepo
  Caller->>IPFSHandler: Request CID
  IPFSHandler->>Database: Resolve CID through pinned_cids
  Database-->>IPFSHandler: Candidate Git OIDs
  IPFSHandler->>VisibilityPack: Compute caller blob or tree allow-set
  VisibilityPack->>GitRepo: Enumerate reachable commits and paths
  GitRepo-->>VisibilityPack: Reachable object paths
  VisibilityPack-->>IPFSHandler: Allowed object OIDs
  IPFSHandler-->>Caller: Serve object or return 404
Loading

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related issues

Possibly related PRs

  • Gitlawb/node#128 — Established per-caller path-scoped gating in the IPFS handler.
  • Gitlawb/node#133 — Introduced reachable caller-aware allow-set gating extended here to trees.
  • Gitlawb/node#90 — Directly relates to the pinned_cids data flow used for CID resolution.

Suggested labels: sev:high, kind:security, subsystem:api

Suggested reviewers: jatmn

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly states the main change: gating GET /ipfs/{cid} tree objects to prevent subtree structure leaks.
Description check ✅ Passed It covers the problem, approach, scope, tests, and issue closure, so the template's key content is present.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/issue-135-ipfs-cid-tree-gate

Comment @coderabbitai help to get the list of available commands.

@beardthelion beardthelion added crate:node gitlawb-node — the serving node and REST API kind:bug Defect fix — wrong or unsafe behavior subsystem:storage Blob/object store, Arweave, IPFS, archives subsystem:visibility Path-scoped visibility and content withholding labels Jul 10, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
crates/gitlawb-node/src/api/ipfs.rs (1)

143-209: 🚀 Performance & Scalability | 🔵 Trivial

The blob/tree gating, memo selection, and fail-closed arms look correct.

One operational note: /ipfs/{cid} is reachable anonymously, and under path-scoped rules each request against an object that exists in a repo triggers a full-history reachability walk (one git ls-tree -rzt per reachable commit, plus the root-tree pass for trees). The memo is request-scoped only, so a spray of valid blob/tree CIDs against a large-history repo re-runs the walk on every request. Consider a bounded cross-request allow-set cache (keyed by repo id + head oid + caller) and/or rate limiting on this route to cap the cost.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/gitlawb-node/src/api/ipfs.rs` around lines 143 - 209, Mitigate
repeated full-history walks in the /ipfs/{cid} path by adding a bounded
cross-request cache for computed blob/tree allow-sets, keyed by repository ID,
current head OID, object type, and caller identity; invalidate or naturally
bypass entries when the head changes. Implement this around
allowed_blob_set_for_caller, allowed_tree_set_for_caller, and the existing memo
lookup, and consider adding rate limiting for anonymous requests to further cap
abuse.
crates/gitlawb-node/src/git/visibility_pack.rs (1)

391-416: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Avoid ARG_MAX in root_tree_pairs.

Passing every reachable commit to git log on argv scales with history size and can fail on very large repos. When that happens, the tree CID path for path-scoped rules skips the repo and the object falls through to a 404. Feed the commits over stdin instead (git log --stdin --no-walk=unsorted --format=%T).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/gitlawb-node/src/git/visibility_pack.rs` around lines 391 - 416,
Update root_tree_pairs to avoid placing all commit IDs in argv: invoke git log
with --stdin alongside --no-walk=unsorted and --format=%T, write the commits
joined by newlines to the child process stdin, and handle stdin/command errors
consistently with the existing context and status checks. Remove the argument
expansion of commits while preserving the existing tree-pair parsing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/gitlawb-node/src/api/ipfs.rs`:
- Around line 143-209: Mitigate repeated full-history walks in the /ipfs/{cid}
path by adding a bounded cross-request cache for computed blob/tree allow-sets,
keyed by repository ID, current head OID, object type, and caller identity;
invalidate or naturally bypass entries when the head changes. Implement this
around allowed_blob_set_for_caller, allowed_tree_set_for_caller, and the
existing memo lookup, and consider adding rate limiting for anonymous requests
to further cap abuse.

In `@crates/gitlawb-node/src/git/visibility_pack.rs`:
- Around line 391-416: Update root_tree_pairs to avoid placing all commit IDs in
argv: invoke git log with --stdin alongside --no-walk=unsorted and --format=%T,
write the commits joined by newlines to the child process stdin, and handle
stdin/command errors consistently with the existing context and status checks.
Remove the argument expansion of commits while preserving the existing tree-pair
parsing.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 335f1ebc-783c-4552-bfd6-ebc5894e4d9a

📥 Commits

Reviewing files that changed from the base of the PR and between 2109d08 and 89d4928.

📒 Files selected for processing (4)
  • crates/gitlawb-node/src/api/ipfs.rs
  • crates/gitlawb-node/src/git/visibility_pack.rs
  • crates/gitlawb-node/src/test_support.rs
  • crates/gitlawb-node/src/visibility.rs

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Findings

  • [P1] Make the CID tests and lookup use the identifier published by the pin path
    crates/gitlawb-node/src/test_support.rs:2064
    These new assertions request cid_for_oid(...), whose multihash is a Git object ID. The real pin path instead stores CID(sha256(raw object content)), and gl ipfs get sends that stored CID back to this handler. Even with SHA-256 Git repositories those values differ because Git hashes "<type> <len>\\0" + content; get_by_cid therefore treats a real pinned CID as a nonexistent OID and returns 404 before this new tree gate runs. Please make the serving lookup use the same CID-to-object mapping as pinning (or make the two identifiers deliberately identical), and cover a CID produced from the fixture object's raw bytes rather than encoding its OID.

  • [P2] Do not pass the complete history as git log arguments
    crates/gitlawb-node/src/git/visibility_pack.rs:395
    root_tree_pairs adds every reachable commit to the process argv. On a long history this exceeds ARG_MAX (about 32k SHA-256 OIDs on a 2 MiB limit), so spawning git log fails; the handler treats that walk error as a denial and returns 404 even to an authorized caller requesting a reachable/root tree. Please complete CodeRabbit's pending root-tree request by feeding commit IDs through git log --stdin or by batching/root-resolving them during the existing per-commit traversal.

t added 7 commits July 10, 2026 10:31
blob_paths becomes the kind==blob slice of a new private object_paths
(one git ls-tree -rzt per reachable commit, emitting (oid, /path, kind)),
so the blob and forthcoming tree visibility gates share one walk and
cannot drift. blob_paths output is byte-identical; all callers unchanged.
The tree analog of allowed_blob_set_for_caller: tree_paths = object_paths'
kind==tree slice plus each reachable commit's root tree at "/" (derived via
<commit>^{tree} over the shared reachable_commits set, since ls-tree never
emits a commit's own root). allowed_tree_set_for_caller keeps the tree oids
visibility ALLOWS the caller at some path. Fail-closed on dangling/unreachable
trees. Not yet wired into get_by_cid (U4).
…135)

GET /ipfs/{cid} served tree objects of a withheld subtree unconditionally
under a path-scoped rule, leaking every child filename and child oid where
get_tree protects them. Gate trees against the caller's allowed-tree-set the
same way blobs are gated (lazy per-repo memo, spawn_blocking, fail-closed);
commits/tags still serve since they expose only root-level metadata the caller
already cleared the "/" gate for.

Tests (RED before this gate, GREEN after; mutation-certified load-bearing):
- withheld subtree tree CID -> 404 for anon, witnessed on RAW body bytes (a
  tree stores child oids as raw bytes that lossy-decode to U+FFFD, so a hex
  contains-check is vacuous); listed reader/owner still see it (200)
- root/ancestor tree, commit, tag stay served; dangling tree fails closed
- visibility_check denies the withheld dir's own path /secret (T1)
Update the get_by_cid and allowed_blob_set_for_caller doc comments and the
CONCEPTS.md "Structural object" entry: trees are now gated on the object-fetch
surface via the allowed-tree-set; only commits/tags remain served-as-structure.
The entry also records the cross-surface gap the fetch gate exposes — the
replication/pin filter still exports withheld-subtree trees (pin-path leak, #172).
… docs

Review follow-up on the #135 tree gate:
- tree_paths computes reachable_commits ONCE and threads it to both the ls-tree
  walk and the root-tree pass (was computing it twice), and resolves all root
  trees in one `git log --no-walk --format=%T` pass instead of a per-commit
  `git rev-parse ^{tree}` fan-out — a tree-set walk now costs the same
  subprocess order as the blob walk.
- Extract the shared allowed-set inner loop (allowed_set_from_pairs).
- The deny test's raw-byte checks on the 404 body were vacuous (the 404 body is
  an opaque error string); move the structure witness to the reader's 200
  response, where the tree body genuinely carries b.txt + the raw child oid.
  Status stays the load-bearing deny check (mutation-recertified RED).
- Fix the module doc that still claimed trees are never withheld.
Add the handler- and helper-level cases the code review flagged:
- ipfs_cid_dangling_tree_fails_closed_under_path_rules: a dangling tree via
  git mktree 404s through cid_router for anon AND owner (handler wiring of the
  fail-closed tree arm, not just the helper).
- ipfs_cid_tree_served_when_no_path_scoped_rule: a tree CID serves 200 when no
  path-scoped rule exists (the skip-walk branch is not over-gating trees).
- allowed_tree_set_includes_tree_shared_across_allowed_and_denied_paths: T2
  content-dedup — a tree oid reachable at both an allowed and a withheld path is
  included (allowed-wins).
- allowed_tree_set_includes_root_trees_of_all_reachable_commits: the batched
  root-tree pass returns every reachable commit's root, not just HEAD's.
#135)

get_by_cid treated the CID's sha2-256 digest as a git oid and cat-file'd it, but a real pin CID digests the raw object content (Cid::from_git_object_bytes), not the framed git object, so every pinned CID 404'd before the #135 tree gate could run. Resolve the incoming CID to its oid through the pinned_cids table (new Db::oid_for_cid + idx_pinned_cids_cid) and gate on that oid; a CID never pinned here is an opaque 404, uniform with a genuine not-found and a visibility denial.

The tree-gate tests now build the request CID the way the pin path does (pin_cid_for: read raw bytes, Cid::from_git_object_bytes, record_pinned_cid) instead of from the oid, so they exercise the gate on a production CID rather than an identifier that never occurs. RED before the serve fix (the served-object assertions 404), GREEN after.

Also feed root_tree_pairs' commit set to 'git log --stdin' on stdin instead of argv: a long history overflowed ARG_MAX, failing the walk, and the caller fail-closed 404s an authorized reader of a reachable/root tree. Oids are written from a separate thread while the main thread drains stdout, so large input and output cannot deadlock on the pipe buffers; a scale test over 2500 commits guards it.

Resolves jatmn's P1 and P2 on #173.
@beardthelion beardthelion force-pushed the fix/issue-135-ipfs-cid-tree-gate branch from 89d4928 to 7dec45c Compare July 10, 2026 15:42
The index was appended to the v1 bundle, which is recorded once in schema_migrations and then skipped, so a node already past v1 would never create it. Move it to a new v11 migration and add an upgrade-path test that drops the index plus its migration record and asserts run_migrations() recreates it.

Follow-up to jatmn's P1/P2 on #173; addresses INV-7 caught in pre-push review.
@beardthelion

Copy link
Copy Markdown
Collaborator Author

Both addressed as of 52b81fd.

P1 (CID → object mapping). You're right that the lookup and the tests diverged from the pin path. get_by_cid was treating the CID's sha2-256 digest as the git oid, but the pin path stores CID(sha256(raw content)), which never equals the oid (git frames the object as "<type> <len>\0" + content before hashing), so a real pinned CID 404'd before the tree gate ever ran. get_by_cid now resolves the incoming CID to its oid through the pinned_cids table (oid_for_cid, backed by an indexed column added as a versioned migration) and gates on that oid; a CID never pinned here is an opaque 404, uniform with a genuine not-found and a visibility denial. The tree-gate tests now build the request CID the way the pin path does (read the object's raw bytes, Cid::from_git_object_bytes, record the pin) rather than encoding the oid, so they exercise the gate on a real production CID. They fail against the old digest-as-oid handler and pass after the fix.

P2 (git log argv). root_tree_pairs now feeds the commit oids to git log --no-walk=unsorted --format=%T --stdin on stdin instead of argv, so a long history can no longer overflow ARG_MAX and fail-close an authorized reader of a reachable/root tree. The oids are written from a separate thread while the main thread drains stdout, so a large input and output can't deadlock on the pipe buffers; a 2500-commit test covers it.

Rebased onto main.

@beardthelion beardthelion requested a review from jatmn July 10, 2026 16:14
… claim (#135)

The test asserted a deadlock guard, but git log --stdin reads its whole revision list to EOF before emitting %T, so the naive write-then-drain form never deadlocks (verified by reverting the writer thread and running at N=40000, ~1.6 MiB each way). Rename and recomment to what it actually verifies: parity at scale plus a liveness watchdog. Test body unchanged.

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Findings

  • [P1] Bound the new tree visibility walk on the public retrieval route
    crates/gitlawb-node/src/api/ipfs.rs:173
    Every request for a known tree CID in a repo with a path-scoped rule recomputes the complete allowed-tree set: rev-list, one recursive ls-tree per reachable commit, and a root-tree pass. The memo only lasts for that one request, while /ipfs/{cid} is anonymous and the public pins index exposes valid CIDs. An unauthenticated caller can therefore repeat a tree-CID request and saturate the blocking pool/CPU with unbounded full-history work. Please add a bounded, revision-aware cross-request cache and/or a route-level work/rate limit before enabling this path.

  • [P2] Try every object recorded for a content CID
    crates/gitlawb-node/src/db/mod.rs:2188
    The new index deliberately permits duplicate CIDs, but this unordered LIMIT 1 chooses only one mapped OID and the handler never tries another. CIDs here hash untyped raw object bytes, so a tree and a blob containing its raw tree bytes have different Git OIDs but the same CID; both can be pinned. If PostgreSQL chooses a withheld, stale, or absent object while another mapped object is readable, the endpoint returns 404 for a valid advertised CID. Return all matching OIDs and run each through the existing repository/visibility checks (with collision coverage) rather than selecting one arbitrarily.

  • [P2] Canonicalize parsed CIDv1 values before the database lookup
    crates/gitlawb-node/src/api/ipfs.rs:92
    CidGeneric::from_str accepts valid CIDv1 multibase encodings, but this lookup uses the original request spelling. Pins are stored under the canonical base32 string produced by Cid::from_git_object_bytes(...).to_string(), so an equivalent base58/base64 CID passes validation yet misses pinned_cids and returns 404. Look up cid.to_string() (or a binary canonical key) and add an alternate-encoding retrieval test.

@beardthelion

Copy link
Copy Markdown
Collaborator Author

Ready for re-review on 1e5035c. Both findings are in, and I re-verified them by execution on this head:

  • P1 (CID identity). get_by_cid resolves the incoming CID to its git oid via pinned_cids (oid_for_cid) and gates on that oid; a never-pinned CID is an opaque 404. The tree-gate tests build the request CID the pin-path way (Cid::from_git_object_bytes), so they run against a real production CID rather than the oid, forcing the gate open flips a withheld subtree from 404 to a served leak.
  • P2 (argv overflow). root_tree_pairs feeds the commit oids through git log --stdin instead of argv, so a long history can no longer overflow ARG_MAX and fail-close an authorized reader; the stdin path returns every root tree at scale.

Ready for another look.

@beardthelion beardthelion requested a review from jatmn July 12, 2026 17:05
…aps (#173)

Resolve jatmn's three CHANGES_REQUESTED findings on GET /ipfs/{cid}, each
verified by execution (revert -> RED, fix -> GREEN):

- [P1] Rate-limit the full-history allowed-set walk per source IP, checked
  once right before the walk spawns (the resource sink), reusing the same
  RateLimiter and trusted-proxy key as the push brake. GITLAWB_IPFS_RATE_LIMIT
  (default 600/hr, 0 disables, bounded key map). A memo hit or a cheap
  non-path-scoped fetch is never braked; the key is the non-farmable client IP,
  not the DID.
- [P2] Resolve a CID to every mapped oid (oids_for_cid) instead of LIMIT 1, and
  try each through the repo/visibility loop, so a withheld or absent duplicate
  no longer false-404s a CID that has a readable object.
- [P2] Canonicalize the parsed CID (cid.to_string()) before the pinned_cids
  lookup so an equivalent base58/base64 spelling resolves to the canonical
  base32 key the pin path stores.

Tests: ipfs_walk_rate_limited_per_source (shed, per-source isolation, and the
must-not on a cheap non-walk fetch), oids_for_cid_returns_all_duplicates,
ipfs_cid_collision_serves_readable_duplicate, ipfs_alt_encoding_cid_resolves.
Full node suite green (502).
@beardthelion

Copy link
Copy Markdown
Collaborator Author

Addressed all three on 2fa6449, each verified by execution (reverting the fix reproduces the finding, the fix clears it):

[P1] Bound the tree-visibility walk. The full-history allowed-set walk is now rate-limited per source IP, checked once immediately before it spawns (the resource sink), reusing the same RateLimiter and trusted-proxy key as the push brake (GITLAWB_IPFS_RATE_LIMIT, default 600/hr, 0 disables, bounded key map). It fires only on a real walk: a request-memo hit or a cheap non-path-scoped fetch is never braked, and the key is the client IP, not the farmable DID. ipfs_walk_rate_limited_per_source covers the shed (2nd walk 429), per-source isolation (a second IP still 200), and the must-not (a public non-walk fetch from the exhausted IP still 200).

[P2] Try every object recorded for a content CID. oid_for_cid's LIMIT 1 is replaced by oids_for_cid, which returns every mapped oid; the handler runs each through the existing repo/visibility loop and serves the first readable one, so a withheld, stale, or absent duplicate no longer 404s a CID that has a readable object. Covered by oids_for_cid_returns_all_duplicates and a collision test that pins an absent oid first and a readable one second under the same CID.

[P2] Canonicalize CIDv1 before the lookup. The lookup keys on cid.to_string() (canonical base32) instead of the request spelling, so an equivalent base58/base64 CID resolves. ipfs_alt_encoding_cid_resolves sends a base58 spelling of a pinned CID.

Full node suite green (502).

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/gitlawb-node/src/api/ipfs.rs (1)

143-159: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Stale invariant comment: "one request builds exactly one of the two sets."

The tree analog (#135): a withheld subtree's tree object is gated the same way a withheld blob is, so its structure cannot leak by CID where get_tree protects it. The adjacent claim that Built lazily and only for a tree fetch (a request is one CID = one object type), so one request builds exactly one of the two sets — no double walk no longer holds: the new multi-candidate oid loop can resolve a single CID to both a blob and a tree oid in the same repo (the documented CID-collision case in db/mod.rs's oids_for_cid), which would populate both allowed_blob_memo and allowed_tree_memo within one request.

Update the comment to reflect that both sets can now be built in the collision case.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/gitlawb-node/src/api/ipfs.rs` around lines 143 - 159, Update the
comment above allowed_blob_memo and allowed_tree_memo to remove the claim that
one request builds exactly one set; state that the sets are built lazily and
that both may be populated when a CID resolves to both blob and tree candidates.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/gitlawb-node/src/api/ipfs.rs`:
- Around line 165-279: The walk_rate_checked guard in the per-object gating flow
only limits the first spawn_blocking walk, allowing subsequent repo/type walks
within one request. Replace the one-time check with per-request walk accounting
and enforce a cap before every allowed_blob_set_for_caller or
allowed_tree_set_for_caller invocation, rejecting or skipping once exhausted
while preserving the existing IP limiter check and fail-closed walk handling.

In `@crates/gitlawb-node/src/main.rs`:
- Around line 328-345: The periodic cleanup task must also invoke cleanup on
ipfs_rate_limiter. Update the cleanup loop to call its cleanup method alongside
the other rate limiters, ensuring expired source-IP entries are removed
regularly and the 200,000-key bound does not retain stale clients.

---

Outside diff comments:
In `@crates/gitlawb-node/src/api/ipfs.rs`:
- Around line 143-159: Update the comment above allowed_blob_memo and
allowed_tree_memo to remove the claim that one request builds exactly one set;
state that the sets are built lazily and that both may be populated when a CID
resolves to both blob and tree candidates.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 06879bbd-f899-4792-9943-5bc38b5533fa

📥 Commits

Reviewing files that changed from the base of the PR and between 1e5035c and 2fa6449.

📒 Files selected for processing (6)
  • crates/gitlawb-node/src/api/ipfs.rs
  • crates/gitlawb-node/src/auth/mod.rs
  • crates/gitlawb-node/src/db/mod.rs
  • crates/gitlawb-node/src/main.rs
  • crates/gitlawb-node/src/state.rs
  • crates/gitlawb-node/src/test_support.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/gitlawb-node/src/test_support.rs

Comment thread crates/gitlawb-node/src/api/ipfs.rs
Comment thread crates/gitlawb-node/src/main.rs

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Findings

  • [P1] Bound every full-history walk, not just the first one in a request
    crates/gitlawb-node/src/api/ipfs.rs:216
    walk_rate_checked consumes one IP quota token and then remains true while the nested CID-candidate × repository loops continue. A shared pinned blob/tree CID can be present in any number of public path-scoped repositories (and a CID can intentionally have multiple OID candidates); when the caller is denied in each, the request starts one allowed_*_set_for_caller full-history walk per repository after paying for only the first. Thus 600 requests per IP can still cause 600 × N rev-list/ls-tree walks and exhaust the blocking pool/CPU. Charge or cap every spawned walk (and add a multi-repository regression) rather than treating the first check as a request-wide authorization for unbounded work.

  • [P1] Do not leave resolvable pinned CIDs on the unbounded all-repository probe path
    crates/gitlawb-node/src/api/ipfs.rs:117
    The new pinned_cids lookup makes the CIDs published by the unsigned pin index reach this loop, but the handler still acquires every readable repository and runs the synchronous git cat-file -t probe before it can determine which repository contains the OID. The IP limiter is only consulted later, after a path-scoped blob/tree is found, so a CID for an old, stale, or unscoped pin can repeatedly force O(repositories) local probes (and cold Tigris existence/download work) without consuming a quota token. This reactivates the availability problem tracked in #164 for the now-functional CID contract; associate pins with their owning repository/rows or apply a route-level bound before scanning, and move the blocking probe off the async runtime.

  • [P2] Include the IPFS limiter in the periodic expiry sweep
    crates/gitlawb-node/src/main.rs:428
    The new limiter has the same client-controlled key space and one-hour window as the other bounded IP limiters, but the cleanup task clones and cleans only the older limiters. RateLimiter::check performs a global expiry sweep only when its map is already full, so a distributed request burst can retain up to 200,000 expired IP windows indefinitely during normal traffic. Clone state.ipfs_rate_limiter here and call cleanup() with the other limiter cleanup calls.

…er (#173)

Two review findings on the /ipfs/{cid} retrieval path.

Bound the full-history walk fan-out per request. The ipfs_rate_limiter check
fires once per request, but within a single request the object can exist under
path-scoped rules in many repos, and each distinct repo pays its own
spawn_blocking allowed-set walk (the memo only dedups the same repo). One
request could therefore fan out to O(repos) walks for a single rate-limiter
token (INV-10). Add MAX_HISTORY_WALKS_PER_REQUEST (16): once that many walks
have run, no further walk is spawned for the rest of the request. This also
closes jatmn's open P1 ("bound every full-history walk, not just the first
one").

The ceiling uses a plain break, not a whole-search break. The budget persists
across the outer oid-candidate loop, so a later candidate servable WITHOUT a
walk (a commit/tag, or a no-rule public copy) is still served, while any
further walk it would need re-trips the guard and is skipped. Breaking the
whole search would 404 that free candidate for no amplification benefit.

Sweep the ipfs limiter in the periodic cleanup task. ipfs_rate_limiter was the
one bounded limiter the 300s cleanup loop never called cleanup() on, so its map
sat full of stale source-IP entries until an inline capacity sweep reclaimed
them at the 200k cap. The six cleanup calls now live in
AppState::sweep_rate_limiters, which the loop drives, so the set is testable and
a new limiter has one place to be added.

Tests (each RED->GREEN verified by execution):
- ipfs_walk_fanout_capped_per_request: cap+1 deniers precede a readable copy;
  capped -> 404, neutralizing the break -> 200.
- ipfs_walk_cap_still_serves_walk_free_candidate: a multi-oid CID whose blob
  candidate burns the budget still serves its walk-free commit candidate (200);
  a whole-search break -> 404.
- sweep_rate_limiters_includes_ipfs_limiter: drives the sweep and asserts the
  ipfs limiter's expired entry is evicted; dropping its cleanup() -> entry
  survives.
@beardthelion beardthelion requested a review from jatmn July 13, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

crate:node gitlawb-node — the serving node and REST API kind:bug Defect fix — wrong or unsafe behavior subsystem:storage Blob/object store, Arweave, IPFS, archives subsystem:visibility Path-scoped visibility and content withholding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GET /ipfs/{cid} serves tree/commit objects of withheld subtrees, leaking structure get_tree protects (KTD3)

2 participants