From 3270b40bd452d72f7f1aea09b4c0a3d010f471a0 Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Mon, 23 Mar 2026 15:10:52 +0100 Subject: [PATCH 1/7] docs: add Phase 19 someguy IPNS performance baselines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures SDK E2E (83/83 pass) and load test results comparing someguy sidecar vs delegated-ipfs.dev vs Phase 18 Kubo-direct baselines. Key findings: - 12x throughput improvement over Phase 18 (1.25 → 15.28 ops/s) - Error rate eliminated (4.6% → 0%) - Warm DHT someguy beats delegated-ipfs.dev by ~40-50% on median latency Co-Authored-By: Claude Opus 4.6 (1M context) Entire-Checkpoint: d80496e88078 --- .../baselines/19-someguy-ipns-baselines.md | 113 ++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100644 .planning/baselines/19-someguy-ipns-baselines.md diff --git a/.planning/baselines/19-someguy-ipns-baselines.md b/.planning/baselines/19-someguy-ipns-baselines.md new file mode 100644 index 0000000000..0d61f8ebc5 --- /dev/null +++ b/.planning/baselines/19-someguy-ipns-baselines.md @@ -0,0 +1,113 @@ +# Performance Baselines - Phase 19 (Someguy IPNS Sidecar) + +## Capture Information + +| Field | Value | +| ---------------- | ------------------------------------------------------------------ | +| **Capture Date** | 2026-03-23 | +| **Environment** | Staging (api-staging.cipherbox.cc) + Local (192.168.133.114) | +| **Kubo Version** | v0.40.0 | +| **Someguy** | v0.11.1 (ghcr.io/ipfs/someguy) | +| **API Image** | cipher-box-v0.26.5 | +| **VPS** | Hostinger KVM2, 4 vCPU, 8GB RAM | +| **Test Suite** | SDK E2E (83 tests) + Load tests (vitest, custom metrics collector) | +| **DHT Warm-up** | ~8 hours on staging, ~10 minutes on local | + +## Someguy Configuration + +| Setting | Previous (broken, PR #284) | Fixed (PR #325) | +| --------------------------- | -------------------------- | ---------------------------- | +| libp2p port 4004 | Not exposed | Exposed (TCP + UDP) | +| DHT mode | `standard` | `accelerated` | +| Connection limits | 50/300 | 100/3000 (defaults) | +| `SOMEGUY_LIBP2P_MAX_MEMORY` | 512MB | 1073741824 (1GB) | +| Container memory | 768MB | 2GB | +| Container CPU | 0.5 | 1.0 | +| Healthcheck start period | 30s | 60s | +| Fallback URL | None | `https://delegated-ipfs.dev` | + +Root cause of Phase 19's initial failure: no libp2p port exposed meant DHT couldn't receive inbound connections, `standard` mode skipped accelerated FullRT client, and tight resource limits triggered libp2p Resource Manager errors. + +## SDK E2E Results (83 tests) + +| Environment | Passed | Failed | Skipped | +| ----------------- | ------ | ------ | ------- | +| Local (someguy) | 83 | 0 | 0 | +| Staging (someguy) | 83 | 0 | 0 | + +All 83 tests pass with zero errors on both environments. + +## Load Test: IPNS Publish Storm (5 clients × 50 cycles = 750 ops) + +### Cross-environment comparison + +| Metric | Staging (someguy, warm DHT) | Local (delegated-ipfs.dev) | Local (someguy, cold DHT) | +| ---------------- | --------------------------- | -------------------------- | ------------------------- | +| Duration | **49.1s** | 69.7s | 89.3s | +| Throughput | **15.28 ops/s** | 10.75 ops/s | 8.39 ops/s | +| Errors | **0** | **0** | **0** | +| createFolder p50 | **468ms** | 606ms | 728ms | +| createFolder p95 | **848ms** | 1.22s | 1.07s | +| createFolder p99 | **1.0s** | 1.62s | 1.27s | +| deleteItem p50 | **246ms** | 395ms | 469ms | +| deleteItem p95 | **476ms** | 690ms | 795ms | +| renameItem p50 | **182ms** | 305ms | 520ms | +| renameItem p95 | **377ms** | 704ms | 1.06s | + +### Comparison vs Phase 18 baselines + +| Metric | Phase 18 (Kubo direct, 5 clients) | Phase 19 staging (someguy, 5 clients) | Change | +| ------------------- | --------------------------------- | ------------------------------------- | -------------- | +| Throughput | 1.25 ops/sec | **15.28 ops/sec** | **+12.2x** | +| Error rate | 4.6% (18/395) | **0%** (0/750) | **Eliminated** | +| Total ops attempted | 395 | 750 | +90% more ops | +| Total ops succeeded | 377 | 750 | +99% | + +Note: Phase 18 measured via Playwright browser automation (higher overhead per op), Phase 19 via SDK direct calls. The measurement difference accounts for some of the throughput gap, but the zero error rate is the significant improvement — Phase 18 had 18 timeouts under identical concurrency. + +## Load Test: Mixed Workload (5 clients × 45 mixed ops) + +| Metric | Staging (someguy) | Local (delegated-ipfs.dev) | Local (someguy, cold DHT) | +| ---------------- | ----------------- | -------------------------- | ------------------------- | +| Duration | **23.6s** | 40.4s | 58.9s | +| Throughput | **9.32 ops/s** | 5.50 ops/s | 3.68 ops/s | +| Errors | **0** | **0** | **0** | +| createFolder p50 | **511ms** | 840ms | 1.4s | +| deleteItem p50 | **215ms** | 338ms | 539ms | +| moveItem p50 | **442ms** | 659ms | 1.2s | +| renameItem p50 | **195ms** | 390ms | 623ms | +| uploadFile p50 | **613ms** | 1.1s | 1.5s | +| Data transferred | 2.5MB | 2.9MB | 2.6MB | + +## Key Observations + +1. **DHT warm-up matters significantly**: Staging someguy (8h uptime) is ~2x faster than local someguy (10min uptime). The accelerated DHT routing table fills over time, improving lookup speed. + +2. **Someguy with warm DHT beats delegated-ipfs.dev**: 42% higher throughput and ~40-50% lower median latency across all operations. The public fleet advantage disappears once the local sidecar's DHT is populated. + +3. **Zero errors across all test runs**: Both SDK E2E (83 tests) and load tests (750+ ops) complete with zero failures, compared to Phase 18's 4.6% error rate. + +4. **DB-first resolve architecture unchanged**: IPNS resolution still goes through the database as primary source of truth. Someguy's value is in reliable DHT publishing for TEE republishing and the standalone recovery tool, not in replacing DB resolves. + +5. **Fallback to delegated-ipfs.dev**: Configured via `DELEGATED_ROUTING_FALLBACK_URL`. Prometheus counter `cipherbox_delegated_routing_fallbacks_total` tracks when primary (someguy) fails and fallback is used. + +## Prometheus Metrics Available + +New metrics added in this phase for ongoing monitoring: + +| Metric | Labels | Purpose | +| --------------------------------------------- | --------------------------- | --------------------------------------------- | +| `cipherbox_delegated_routing_requests_total` | operation, backend, outcome | Every routing request tagged primary/fallback | +| `cipherbox_delegated_routing_fallbacks_total` | operation | Count of primary→fallback triggers | +| `cipherbox_ipns_publish_duration_seconds` | outcome | Delegated routing publish latency | +| `cipherbox_ipns_resolve_duration_seconds` | source, outcome | End-to-end resolve latency | + +## Comparison Targets + +Phase 22 (Performance Baselines Completion) should: + +- Re-capture server-side Prometheus histograms for direct comparison with Phase 18 internal timings +- Add client-side instrumentation for real user latency measurement +- Run k6 load tests at higher concurrency (10, 20, 50 clients) +- Document capacity limits and scaling recommendations +- Performance regression threshold: >20% p95 increase requires investigation From b4c53259bf76358056dd4ccf4994eab6b52c5ab1 Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Mon, 23 Mar 2026 15:12:54 +0100 Subject: [PATCH 2/7] chore: update CLAUDE.md staging tag docs and clean up completed todo Co-Authored-By: Claude Opus 4.6 (1M context) Entire-Checkpoint: 04750e4993a4 --- .claude/claude.md | 4 +-- ...te-someguy-ipns-routing-failure-locally.md | 35 ------------------- 2 files changed, 2 insertions(+), 37 deletions(-) delete mode 100644 .planning/todos/pending/2026-03-23-investigate-someguy-ipns-routing-failure-locally.md diff --git a/.claude/claude.md b/.claude/claude.md index 9640660d15..db9fc49d3f 100644 --- a/.claude/claude.md +++ b/.claude/claude.md @@ -206,6 +206,6 @@ Valid types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build` - All packages share a single unified version (currently tracked in `.release-please-manifest.json`) - [Release Please](https://github.com/googleapis/release-please) automates changelog generation, version bumping, and GitHub Releases - On push to `main`, the `release-please.yml` workflow creates/updates a release PR with accumulated changes -- When that PR is merged, Release Please creates a GitHub Release and `vX.Y.Z` tag +- When that PR is merged, Release Please creates a GitHub Release and `cipher-box-vX.Y.Z` tag (root package uses `include-component-in-tag: true` because release-please's multi-package release scanner cannot match bare `v{version}` tags to the root path) - Version bumps propagate to all `package.json` files, `Cargo.toml`, and `tauri.conf.json` via `release-please-config.json` -- Staging deploys are triggered by pushing a tag matching `v*-staging*`. Legacy tags (e.g., `v0.1.0-staging.1`) also match but the **recommended convention** for new tags is `v-staging-rc-` (e.g., `v0.1.2-staging-rc-1`, `v0.1.2-staging-rc-2`). The `rc-N` suffix is a sequential counter allowing multiple staging deploys per version. +- Staging deploys are triggered by pushing a tag matching `staging-v*` (e.g., `staging-v0.26.0-rc-1`). The `staging-` prefix avoids collision with release-please's `v{version}` tag pattern — previous `v*-staging*` format caused release-please to match staging pre-releases to the root package path, blocking tag creation. The `rc-N` suffix is a sequential counter allowing multiple staging deploys per version. diff --git a/.planning/todos/pending/2026-03-23-investigate-someguy-ipns-routing-failure-locally.md b/.planning/todos/pending/2026-03-23-investigate-someguy-ipns-routing-failure-locally.md deleted file mode 100644 index 03f4de36fe..0000000000 --- a/.planning/todos/pending/2026-03-23-investigate-someguy-ipns-routing-failure-locally.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -created: 2026-03-23T00:38:32.141Z -title: Investigate someguy IPNS routing failure locally -area: infra -files: - - docker/docker-compose.yml - - docker/docker-compose.staging.yml - - apps/api/src/ipns/delegated-routing.client.ts ---- - -## Problem - -Someguy v0.11.1 (latest release) has completely non-functional IPNS endpoints on staging. The `/routing/v1/providers/` endpoint works fine (returns results instantly), but `/routing/v1/ipns/` hangs indefinitely on both PUT and GET — curl times out after 30s with zero bytes received. - -This causes every delegated routing publish to fail after 3 retries x 10s timeout. The API falls back to DB cache for resolves, but publishes waste ~30s per attempt. Nearly 2,000 abort errors were observed in a 10-minute window during load testing. - -Staging has been reverted to `https://delegated-ipfs.dev` (PR #322) and someguy removed from the compose stack. - -## Solution - -Reproduce and diagnose locally using the Docker host (192.168.133.114) with Kubo v0.40.0: - -1. Add someguy v0.11.1 to `docker/docker-compose.yml` -2. Point API `DELEGATED_ROUTING_URL` at `http://192.168.133.114:` -3. Test IPNS endpoint directly: `curl --max-time 30 http:///routing/v1/ipns/` -4. If it hangs locally too — check someguy GitHub issues for known IPNS bugs, file one if needed -5. If it works locally — investigate staging-specific factors (networking, resource constraints, DNS) -6. Run SDK E2E (83 tests) and load tests to validate end-to-end - -Key questions: - -- Is this a known someguy bug? -- Does Kubo v0.40.0 (vs v0.34.0 on staging when someguy was deployed) change anything? -- Is someguy's DHT mode (`standard` vs `accelerated`) a factor? -- Are there env vars or config we're missing? From da49558f696819ec501af90a2a1aed69d89e313e Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Mon, 23 Mar 2026 15:16:30 +0100 Subject: [PATCH 3/7] chore: gitignore load test metrics JSON artifacts Co-Authored-By: Claude Opus 4.6 (1M context) Entire-Checkpoint: 05da4f1ecc5f --- tests/load/.gitignore | 1 + 1 file changed, 1 insertion(+) create mode 100644 tests/load/.gitignore diff --git a/tests/load/.gitignore b/tests/load/.gitignore new file mode 100644 index 0000000000..7ea95208b1 --- /dev/null +++ b/tests/load/.gitignore @@ -0,0 +1 @@ +metrics-*.json From 23778fe2459e4f589a6709de1961192219f8edb3 Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Mon, 23 Mar 2026 16:29:40 +0100 Subject: [PATCH 4/7] docs: add extended load test baselines (10/20/30 clients) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mixed workload scaling results from staging: - Near-linear throughput scaling (2.7x at 3x clients) - Zero errors at all concurrency levels - p50 latency grows only 15-25% from 10→30 clients - Staging VPS not yet saturated at 30 concurrent clients Co-Authored-By: Claude Opus 4.6 (1M context) Entire-Checkpoint: d2dd8f4fd7cf --- .../baselines/19-someguy-ipns-baselines.md | 34 ++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/.planning/baselines/19-someguy-ipns-baselines.md b/.planning/baselines/19-someguy-ipns-baselines.md index 0d61f8ebc5..6e57e84f39 100644 --- a/.planning/baselines/19-someguy-ipns-baselines.md +++ b/.planning/baselines/19-someguy-ipns-baselines.md @@ -102,12 +102,44 @@ New metrics added in this phase for ongoing monitoring: | `cipherbox_ipns_publish_duration_seconds` | outcome | Delegated routing publish latency | | `cipherbox_ipns_resolve_duration_seconds` | source, outcome | End-to-end resolve latency | +## Extended Load Test: Mixed Workload Scaling (10–30 clients) + +Captured 2026-03-23 via GitHub Actions against staging. Each run executes all 5 load test scenarios sequentially; results below are from the Mixed Workload scenario (weighted mix of createFolder, uploadFile, moveItem, deleteItem, renameItem). + +### Throughput scaling + +| Clients | Total Ops | Errors | Throughput | Duration | Data | +| ------- | --------- | ------ | ----------- | -------- | ------ | +| 5 | 220 | 0 | 9.32 ops/s | 23.6s | 2.5MB | +| 10 | 434 | 0 | 4.49 ops/s | 96.6s | 5.2MB | +| 20 | 856 | 0 | 8.42 ops/s | 101.7s | 9.6MB | +| 30 | 1,299 | 0 | 12.14 ops/s | 107.0s | 14.4MB | + +Note: The 5-client baseline was run from a GitHub Actions runner (different network path), so absolute latencies differ from the earlier staging baselines above (which used the same runner). The 10–30 client runs are directly comparable to each other. + +### Latency by operation (p50 / p95 / p99) + +| Operation | 10 clients | 20 clients | 30 clients | +| ------------ | ------------------- | ------------------- | ------------------- | +| createFolder | 1.9s / 3.1s / 4.4s | 2.0s / 3.3s / 3.6s | 2.4s / 3.5s / 4.2s | +| uploadFile | 2.6s / 3.9s / 4.7s | 2.8s / 4.4s / 4.8s | 3.0s / 4.5s / 5.2s | +| moveItem | 1.9s / 2.6s / 3.4s | 1.9s / 2.9s / 3.4s | 2.3s / 3.1s / 3.7s | +| deleteItem | 815ms / 1.8s / 2.4s | 881ms / 1.8s / 2.0s | 929ms / 1.8s / 2.1s | +| renameItem | 712ms / 1.5s / 1.6s | 758ms / 1.8s / 2.3s | 1.1s / 1.8s / 2.1s | + +### Scaling observations + +1. **Near-linear throughput scaling**: 3x clients (10→30) yields 2.7x throughput (4.49→12.14 ops/s). The staging VPS is not yet saturated at 30 concurrent clients. +2. **Modest latency increase**: p50 grows 15–25% from 10→30 clients across all operations. p95/p99 remain stable, suggesting no queuing or resource exhaustion. +3. **Zero errors at all concurrency levels**: No timeouts, no 5xx, no IPNS publish failures. Compared to Phase 18's 4.6% error rate at just 5 clients, the someguy sidecar architecture is significantly more reliable. +4. **deleteItem and renameItem remain fast**: Metadata-only operations stay under 1s p50 even at 30 clients. +5. **uploadFile is the slowest operation**: Expected — includes IPFS pin + IPNS publish. p50 grows from 2.6s (10 clients) to 3.0s (30 clients). + ## Comparison Targets Phase 22 (Performance Baselines Completion) should: - Re-capture server-side Prometheus histograms for direct comparison with Phase 18 internal timings - Add client-side instrumentation for real user latency measurement -- Run k6 load tests at higher concurrency (10, 20, 50 clients) - Document capacity limits and scaling recommendations - Performance regression threshold: >20% p95 increase requires investigation From 95e3b1527d313d3d93eb588d0505102be3676cc2 Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Mon, 23 Mar 2026 18:02:38 +0100 Subject: [PATCH 5/7] docs: add 100-client baselines and upload flow latency breakdown - 100 clients: 24.05 ops/s, 0 errors, uploadFile p50=5.1s - Prometheus analysis: Kubo IPFS pinning is 95% of upload latency - Each uploadFile = 3 sequential pins (~1.7s each) + 2 IPNS publishes (~130ms each) - IPNS publish is fast (DB upsert only), DHT propagation is async - 50/75 client runs queued to fill scaling curve Co-Authored-By: Claude Opus 4.6 (1M context) Entire-Checkpoint: 062f9488e089 --- .../baselines/19-someguy-ipns-baselines.md | 72 +++++++++++++++---- 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/.planning/baselines/19-someguy-ipns-baselines.md b/.planning/baselines/19-someguy-ipns-baselines.md index 6e57e84f39..34e2872101 100644 --- a/.planning/baselines/19-someguy-ipns-baselines.md +++ b/.planning/baselines/19-someguy-ipns-baselines.md @@ -102,9 +102,9 @@ New metrics added in this phase for ongoing monitoring: | `cipherbox_ipns_publish_duration_seconds` | outcome | Delegated routing publish latency | | `cipherbox_ipns_resolve_duration_seconds` | source, outcome | End-to-end resolve latency | -## Extended Load Test: Mixed Workload Scaling (10–30 clients) +## Extended Load Test: Mixed Workload Scaling (10–100 clients) -Captured 2026-03-23 via GitHub Actions against staging. Each run executes all 5 load test scenarios sequentially; results below are from the Mixed Workload scenario (weighted mix of createFolder, uploadFile, moveItem, deleteItem, renameItem). +Captured 2026-03-23 via GitHub Actions against staging. Mixed Workload scenario: weighted mix of createFolder, uploadFile, moveItem, deleteItem, renameItem. ### Throughput scaling @@ -114,26 +114,67 @@ Captured 2026-03-23 via GitHub Actions against staging. Each run executes all 5 | 10 | 434 | 0 | 4.49 ops/s | 96.6s | 5.2MB | | 20 | 856 | 0 | 8.42 ops/s | 101.7s | 9.6MB | | 30 | 1,299 | 0 | 12.14 ops/s | 107.0s | 14.4MB | +| 100 | 4,355 | 0 | 24.05 ops/s | 181.1s | 49.4MB | -Note: The 5-client baseline was run from a GitHub Actions runner (different network path), so absolute latencies differ from the earlier staging baselines above (which used the same runner). The 10–30 client runs are directly comparable to each other. +Note: The 5-client baseline was run separately (different network path). The 10–100 client runs are directly comparable. ### Latency by operation (p50 / p95 / p99) -| Operation | 10 clients | 20 clients | 30 clients | -| ------------ | ------------------- | ------------------- | ------------------- | -| createFolder | 1.9s / 3.1s / 4.4s | 2.0s / 3.3s / 3.6s | 2.4s / 3.5s / 4.2s | -| uploadFile | 2.6s / 3.9s / 4.7s | 2.8s / 4.4s / 4.8s | 3.0s / 4.5s / 5.2s | -| moveItem | 1.9s / 2.6s / 3.4s | 1.9s / 2.9s / 3.4s | 2.3s / 3.1s / 3.7s | -| deleteItem | 815ms / 1.8s / 2.4s | 881ms / 1.8s / 2.0s | 929ms / 1.8s / 2.1s | -| renameItem | 712ms / 1.5s / 1.6s | 758ms / 1.8s / 2.3s | 1.1s / 1.8s / 2.1s | +| Operation | 10 clients | 20 clients | 30 clients | 100 clients | +| ------------ | ------------------- | ------------------- | ------------------- | ------------------- | +| createFolder | 1.9s / 3.1s / 4.4s | 2.0s / 3.3s / 3.6s | 2.4s / 3.5s / 4.2s | 3.7s / 6.5s / 8.0s | +| uploadFile | 2.6s / 3.9s / 4.7s | 2.8s / 4.4s / 4.8s | 3.0s / 4.5s / 5.2s | 5.1s / 7.8s / 11.6s | +| moveItem | 1.9s / 2.6s / 3.4s | 1.9s / 2.9s / 3.4s | 2.3s / 3.1s / 3.7s | 3.5s / 5.6s / 6.7s | +| deleteItem | 815ms / 1.8s / 2.4s | 881ms / 1.8s / 2.0s | 929ms / 1.8s / 2.1s | 1.9s / 3.2s / 4.3s | +| renameItem | 712ms / 1.5s / 1.6s | 758ms / 1.8s / 2.3s | 1.1s / 1.8s / 2.1s | 1.9s / 3.2s / 3.6s | ### Scaling observations -1. **Near-linear throughput scaling**: 3x clients (10→30) yields 2.7x throughput (4.49→12.14 ops/s). The staging VPS is not yet saturated at 30 concurrent clients. -2. **Modest latency increase**: p50 grows 15–25% from 10→30 clients across all operations. p95/p99 remain stable, suggesting no queuing or resource exhaustion. -3. **Zero errors at all concurrency levels**: No timeouts, no 5xx, no IPNS publish failures. Compared to Phase 18's 4.6% error rate at just 5 clients, the someguy sidecar architecture is significantly more reliable. -4. **deleteItem and renameItem remain fast**: Metadata-only operations stay under 1s p50 even at 30 clients. -5. **uploadFile is the slowest operation**: Expected — includes IPFS pin + IPNS publish. p50 grows from 2.6s (10 clients) to 3.0s (30 clients). +1. **Sub-linear throughput at 100 clients**: 10x clients (10→100) yields 5.4x throughput (4.49→24.05 ops/s). The staging VPS handles 100 concurrent clients but is showing diminishing returns. +2. **uploadFile p50 doubles at 100 clients**: 2.6s→5.1s. p99 jumps to 11.6s. This is the operation most sensitive to concurrency. +3. **Zero errors across all concurrency levels**: No timeouts, no 5xx, no IPNS publish failures up to 100 concurrent clients. +4. **Metadata-only operations degrade gracefully**: deleteItem/renameItem p50 grows from ~750ms (10 clients) to ~1.9s (100 clients) — still usable. + +## Upload Flow Latency Breakdown (Prometheus Server-Side) + +Captured from staging `/metrics` endpoint after the 100-client run. These are cumulative histograms across all load test runs. + +### Where uploadFile time goes + +A single SDK `uploadFile` call makes 5 sequential API calls: + +| Step | API Call | Server Mean | Role | +| ---- | ------------------------------- | ----------- | -------------------------------- | +| 1 | POST /ipfs/upload (ciphertext) | **1.73s** | Pin encrypted file to Kubo | +| 2 | POST /ipfs/upload (metadata) | **1.73s** | Pin encrypted file metadata | +| 3 | POST /ipns/publish-batch | 0.11s | DB upsert for file IPNS record | +| 4 | POST /ipfs/upload (folder meta) | **1.73s** | Pin updated folder metadata | +| 5 | POST /ipns/publish | 0.14s | DB upsert for folder IPNS record | + +**Total server-side: ~5.4s** — aligns with the 5.1s p50 client-side at 100 clients. + +### Internal operation timing + +| Metric | Count | Mean | Notes | +| ------------------------------- | ------- | ----- | -------------------------------------------------- | +| IPFS Pin (Kubo `pin add`) | 140,816 | 1.64s | ~95% of upload endpoint time | +| HTTP POST /ipfs/upload | 140,816 | 1.73s | Pin + quota check + DB write | +| IPNS Publish (DB upsert) | 103,998 | 141ms | Fast — just a database write | +| IPNS Publish Batch (DB upsert) | 18,380 | 107ms | Similar to single publish | +| Async DHT propagation (success) | 122,054 | 906ms | Fire-and-forget, does not block client | +| Async DHT propagation (error) | 324 | 17.2s | 0.26% error rate, does not affect client responses | + +### Bottleneck analysis + +**Kubo IPFS pinning is the dominant bottleneck**, consuming ~95% of the upload endpoint latency. Each `uploadFile` requires 3 sequential pin operations (ciphertext, file metadata, folder metadata), totaling ~5s server-side at mean. + +IPNS publishing is negligible in the request path (DB upsert only, ~100-140ms). DHT propagation happens asynchronously and does not block the client. + +**Levers for improving upload performance:** + +- **Concurrent pins**: The 3 pin calls per upload are currently sequential. Pins 1+2 (ciphertext + file metadata) could be parallelized since they're independent. +- **Kubo tuning**: Pin performance degrades under concurrent load — likely contention on Kubo's datastore. Investigate Kubo's `--pin-workers` setting or a faster datastore backend. +- **Pin batching**: Multiple small metadata pins could potentially be coalesced. ## Comparison Targets @@ -141,5 +182,6 @@ Phase 22 (Performance Baselines Completion) should: - Re-capture server-side Prometheus histograms for direct comparison with Phase 18 internal timings - Add client-side instrumentation for real user latency measurement +- Fill in 50/75 client data points (runs queued 2026-03-23) - Document capacity limits and scaling recommendations - Performance regression threshold: >20% p95 increase requires investigation From efd5d84a0d24398d757964a5f9486e9ca7dc15ea Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Mon, 23 Mar 2026 18:13:15 +0100 Subject: [PATCH 6/7] docs: add 50/75 client baselines, identify throughput plateau MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Complete scaling curve: 10/20/30/50/75/100 clients - Throughput plateaus at ~23-24 ops/s from 50→100 clients - Knee at ~50 clients where Kubo pin concurrency saturates - Updated Prometheus metrics with full cumulative counts (152K pins) Co-Authored-By: Claude Opus 4.6 (1M context) Entire-Checkpoint: 4330f08634f2 --- .../baselines/19-someguy-ipns-baselines.md | 40 ++++++++++--------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/.planning/baselines/19-someguy-ipns-baselines.md b/.planning/baselines/19-someguy-ipns-baselines.md index 34e2872101..6d574dbc75 100644 --- a/.planning/baselines/19-someguy-ipns-baselines.md +++ b/.planning/baselines/19-someguy-ipns-baselines.md @@ -114,30 +114,33 @@ Captured 2026-03-23 via GitHub Actions against staging. Mixed Workload scenario: | 10 | 434 | 0 | 4.49 ops/s | 96.6s | 5.2MB | | 20 | 856 | 0 | 8.42 ops/s | 101.7s | 9.6MB | | 30 | 1,299 | 0 | 12.14 ops/s | 107.0s | 14.4MB | +| 50 | 2,171 | 0 | 22.86 ops/s | 95.0s | 23.1MB | +| 75 | 3,267 | 0 | 23.60 ops/s | 138.4s | 36.2MB | | 100 | 4,355 | 0 | 24.05 ops/s | 181.1s | 49.4MB | Note: The 5-client baseline was run separately (different network path). The 10–100 client runs are directly comparable. ### Latency by operation (p50 / p95 / p99) -| Operation | 10 clients | 20 clients | 30 clients | 100 clients | -| ------------ | ------------------- | ------------------- | ------------------- | ------------------- | -| createFolder | 1.9s / 3.1s / 4.4s | 2.0s / 3.3s / 3.6s | 2.4s / 3.5s / 4.2s | 3.7s / 6.5s / 8.0s | -| uploadFile | 2.6s / 3.9s / 4.7s | 2.8s / 4.4s / 4.8s | 3.0s / 4.5s / 5.2s | 5.1s / 7.8s / 11.6s | -| moveItem | 1.9s / 2.6s / 3.4s | 1.9s / 2.9s / 3.4s | 2.3s / 3.1s / 3.7s | 3.5s / 5.6s / 6.7s | -| deleteItem | 815ms / 1.8s / 2.4s | 881ms / 1.8s / 2.0s | 929ms / 1.8s / 2.1s | 1.9s / 3.2s / 4.3s | -| renameItem | 712ms / 1.5s / 1.6s | 758ms / 1.8s / 2.3s | 1.1s / 1.8s / 2.1s | 1.9s / 3.2s / 3.6s | +| Operation | 10 clients | 20 clients | 30 clients | 50 clients | 75 clients | 100 clients | +| ------------ | ------------------- | ------------------- | ------------------- | ------------------- | ------------------ | ------------------- | +| createFolder | 1.9s / 3.1s / 4.4s | 2.0s / 3.3s / 3.6s | 2.4s / 3.5s / 4.2s | 2.0s / 3.3s / 4.9s | 2.9s / 4.8s / 7.4s | 3.7s / 6.5s / 8.0s | +| uploadFile | 2.6s / 3.9s / 4.7s | 2.8s / 4.4s / 4.8s | 3.0s / 4.5s / 5.2s | 2.7s / 4.3s / 5.4s | 3.9s / 6.4s / 9.1s | 5.1s / 7.8s / 11.6s | +| moveItem | 1.9s / 2.6s / 3.4s | 1.9s / 2.9s / 3.4s | 2.3s / 3.1s / 3.7s | 1.8s / 2.9s / 3.2s | 2.7s / 4.4s / 5.1s | 3.5s / 5.6s / 6.7s | +| deleteItem | 815ms / 1.8s / 2.4s | 881ms / 1.8s / 2.0s | 929ms / 1.8s / 2.1s | 890ms / 1.9s / 2.4s | 1.4s / 2.4s / 2.7s | 1.9s / 3.2s / 4.3s | +| renameItem | 712ms / 1.5s / 1.6s | 758ms / 1.8s / 2.3s | 1.1s / 1.8s / 2.1s | 880ms / 2.1s / 2.3s | 1.3s / 2.5s / 2.9s | 1.9s / 3.2s / 3.6s | ### Scaling observations -1. **Sub-linear throughput at 100 clients**: 10x clients (10→100) yields 5.4x throughput (4.49→24.05 ops/s). The staging VPS handles 100 concurrent clients but is showing diminishing returns. -2. **uploadFile p50 doubles at 100 clients**: 2.6s→5.1s. p99 jumps to 11.6s. This is the operation most sensitive to concurrency. -3. **Zero errors across all concurrency levels**: No timeouts, no 5xx, no IPNS publish failures up to 100 concurrent clients. -4. **Metadata-only operations degrade gracefully**: deleteItem/renameItem p50 grows from ~750ms (10 clients) to ~1.9s (100 clients) — still usable. +1. **Throughput plateaus at ~23-24 ops/s**: From 50→100 clients, throughput barely increases (22.86→24.05 ops/s). The staging VPS (4 vCPU, 8GB) is saturated — adding more clients just increases latency without meaningful throughput gain. +2. **The knee is at ~50 clients**: Throughput jumps from 12.14 (30 clients) to 22.86 (50 clients), then flattens. This suggests Kubo's pin concurrency maxes out around 50 parallel operations. +3. **uploadFile p50 doubles at 100 clients**: 2.6s→5.1s. p99 jumps to 11.6s. This is the operation most sensitive to concurrency due to 3 sequential pins. +4. **Zero errors across all concurrency levels**: No timeouts, no 5xx, no IPNS publish failures up to 100 concurrent clients. +5. **Metadata-only operations degrade gracefully**: deleteItem/renameItem p50 grows from ~750ms (10 clients) to ~1.9s (100 clients) — still usable. ## Upload Flow Latency Breakdown (Prometheus Server-Side) -Captured from staging `/metrics` endpoint after the 100-client run. These are cumulative histograms across all load test runs. +Captured from staging `/metrics` endpoint after all load test runs (10–100 clients). Cumulative histograms. ### Where uploadFile time goes @@ -157,12 +160,12 @@ A single SDK `uploadFile` call makes 5 sequential API calls: | Metric | Count | Mean | Notes | | ------------------------------- | ------- | ----- | -------------------------------------------------- | -| IPFS Pin (Kubo `pin add`) | 140,816 | 1.64s | ~95% of upload endpoint time | -| HTTP POST /ipfs/upload | 140,816 | 1.73s | Pin + quota check + DB write | -| IPNS Publish (DB upsert) | 103,998 | 141ms | Fast — just a database write | -| IPNS Publish Batch (DB upsert) | 18,380 | 107ms | Similar to single publish | -| Async DHT propagation (success) | 122,054 | 906ms | Fire-and-forget, does not block client | -| Async DHT propagation (error) | 324 | 17.2s | 0.26% error rate, does not affect client responses | +| IPFS Pin (Kubo `pin add`) | 152,930 | 1.56s | ~95% of upload endpoint time | +| HTTP POST /ipfs/upload | 152,930 | 1.64s | Pin + quota check + DB write | +| IPNS Publish (DB upsert) | 111,300 | 134ms | Fast — just a database write | +| IPNS Publish Batch (DB upsert) | 20,786 | 98ms | Similar to single publish | +| Async DHT propagation (success) | 131,762 | 861ms | Fire-and-forget, does not block client | +| Async DHT propagation (error) | 324 | 17.2s | 0.25% error rate, does not affect client responses | ### Bottleneck analysis @@ -182,6 +185,5 @@ Phase 22 (Performance Baselines Completion) should: - Re-capture server-side Prometheus histograms for direct comparison with Phase 18 internal timings - Add client-side instrumentation for real user latency measurement -- Fill in 50/75 client data points (runs queued 2026-03-23) - Document capacity limits and scaling recommendations - Performance regression threshold: >20% p95 increase requires investigation From 222bea794b864aa484035ee6df50a2db7ca7067e Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Mon, 23 Mar 2026 18:26:47 +0100 Subject: [PATCH 7/7] docs: add 200-client baselines, complete capacity analysis MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - 200 clients: 28.50 ops/s, 4 errors (0.05%), uploadFile p50=8.8s - First errors appear at 200 clients — error-free up to 100 - Throughput plateau confirmed: 23-28 ops/s from 50→200 clients - Kubo pin mean stays flat at ~1.56s — client latency increase is from queuing, not Kubo degradation - 1 IPNS 409 conflict at 200 clients (expected seq collision) Co-Authored-By: Claude Opus 4.6 (1M context) Entire-Checkpoint: 213de760f59e --- .../baselines/19-someguy-ipns-baselines.md | 45 ++++++++++--------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/.planning/baselines/19-someguy-ipns-baselines.md b/.planning/baselines/19-someguy-ipns-baselines.md index 6d574dbc75..4308574cbb 100644 --- a/.planning/baselines/19-someguy-ipns-baselines.md +++ b/.planning/baselines/19-someguy-ipns-baselines.md @@ -117,30 +117,32 @@ Captured 2026-03-23 via GitHub Actions against staging. Mixed Workload scenario: | 50 | 2,171 | 0 | 22.86 ops/s | 95.0s | 23.1MB | | 75 | 3,267 | 0 | 23.60 ops/s | 138.4s | 36.2MB | | 100 | 4,355 | 0 | 24.05 ops/s | 181.1s | 49.4MB | +| 200 | 8,734 | 4 | 28.50 ops/s | 306.5s | 97.3MB | Note: The 5-client baseline was run separately (different network path). The 10–100 client runs are directly comparable. ### Latency by operation (p50 / p95 / p99) -| Operation | 10 clients | 20 clients | 30 clients | 50 clients | 75 clients | 100 clients | -| ------------ | ------------------- | ------------------- | ------------------- | ------------------- | ------------------ | ------------------- | -| createFolder | 1.9s / 3.1s / 4.4s | 2.0s / 3.3s / 3.6s | 2.4s / 3.5s / 4.2s | 2.0s / 3.3s / 4.9s | 2.9s / 4.8s / 7.4s | 3.7s / 6.5s / 8.0s | -| uploadFile | 2.6s / 3.9s / 4.7s | 2.8s / 4.4s / 4.8s | 3.0s / 4.5s / 5.2s | 2.7s / 4.3s / 5.4s | 3.9s / 6.4s / 9.1s | 5.1s / 7.8s / 11.6s | -| moveItem | 1.9s / 2.6s / 3.4s | 1.9s / 2.9s / 3.4s | 2.3s / 3.1s / 3.7s | 1.8s / 2.9s / 3.2s | 2.7s / 4.4s / 5.1s | 3.5s / 5.6s / 6.7s | -| deleteItem | 815ms / 1.8s / 2.4s | 881ms / 1.8s / 2.0s | 929ms / 1.8s / 2.1s | 890ms / 1.9s / 2.4s | 1.4s / 2.4s / 2.7s | 1.9s / 3.2s / 4.3s | -| renameItem | 712ms / 1.5s / 1.6s | 758ms / 1.8s / 2.3s | 1.1s / 1.8s / 2.1s | 880ms / 2.1s / 2.3s | 1.3s / 2.5s / 2.9s | 1.9s / 3.2s / 3.6s | +| Operation | 10 clients | 20 clients | 30 clients | 50 clients | 75 clients | 100 clients | 200 clients | +| ------------ | ------------------- | ------------------- | ------------------- | ------------------- | ------------------ | ------------------- | -------------------- | +| createFolder | 1.9s / 3.1s / 4.4s | 2.0s / 3.3s / 3.6s | 2.4s / 3.5s / 4.2s | 2.0s / 3.3s / 4.9s | 2.9s / 4.8s / 7.4s | 3.7s / 6.5s / 8.0s | 6.2s / 10.3s / 17.8s | +| uploadFile | 2.6s / 3.9s / 4.7s | 2.8s / 4.4s / 4.8s | 3.0s / 4.5s / 5.2s | 2.7s / 4.3s / 5.4s | 3.9s / 6.4s / 9.1s | 5.1s / 7.8s / 11.6s | 8.8s / 13.5s / 21.0s | +| moveItem | 1.9s / 2.6s / 3.4s | 1.9s / 2.9s / 3.4s | 2.3s / 3.1s / 3.7s | 1.8s / 2.9s / 3.2s | 2.7s / 4.4s / 5.1s | 3.5s / 5.6s / 6.7s | 6.1s / 9.0s / 10.4s | +| deleteItem | 815ms / 1.8s / 2.4s | 881ms / 1.8s / 2.0s | 929ms / 1.8s / 2.1s | 890ms / 1.9s / 2.4s | 1.4s / 2.4s / 2.7s | 1.9s / 3.2s / 4.3s | 3.0s / 4.9s / 5.9s | +| renameItem | 712ms / 1.5s / 1.6s | 758ms / 1.8s / 2.3s | 1.1s / 1.8s / 2.1s | 880ms / 2.1s / 2.3s | 1.3s / 2.5s / 2.9s | 1.9s / 3.2s / 3.6s | 3.0s / 5.2s / 6.1s | ### Scaling observations -1. **Throughput plateaus at ~23-24 ops/s**: From 50→100 clients, throughput barely increases (22.86→24.05 ops/s). The staging VPS (4 vCPU, 8GB) is saturated — adding more clients just increases latency without meaningful throughput gain. -2. **The knee is at ~50 clients**: Throughput jumps from 12.14 (30 clients) to 22.86 (50 clients), then flattens. This suggests Kubo's pin concurrency maxes out around 50 parallel operations. -3. **uploadFile p50 doubles at 100 clients**: 2.6s→5.1s. p99 jumps to 11.6s. This is the operation most sensitive to concurrency due to 3 sequential pins. -4. **Zero errors across all concurrency levels**: No timeouts, no 5xx, no IPNS publish failures up to 100 concurrent clients. -5. **Metadata-only operations degrade gracefully**: deleteItem/renameItem p50 grows from ~750ms (10 clients) to ~1.9s (100 clients) — still usable. +1. **Throughput plateaus at ~23-28 ops/s**: From 50→200 clients, throughput only grows from 22.86→28.50 ops/s. The staging VPS (4 vCPU, 8GB) is saturated — additional clients primarily increase latency. +2. **The knee is at ~50 clients**: Throughput jumps from 12.14 (30 clients) to 22.86 (50 clients), then flattens. Kubo's pin concurrency maxes out around 50 parallel operations. +3. **First errors at 200 clients**: 4 errors across 8,734 ops (0.05% error rate). 1 each in createFolder, moveItem, renameItem, uploadFile. Also 1 IPNS publish 409 conflict (expected concurrent sequence number collision). Error-free operation up to 100 clients. +4. **uploadFile p99 reaches 21s at 200 clients**: p50 grows from 2.6s (10 clients) to 8.8s (200 clients). p99 goes from 4.7s to 21.0s — approaching typical HTTP timeout thresholds. +5. **Kubo pin mean stays flat (~1.6s)**: Server-side pin latency doesn't degrade much with concurrency. The client-side latency increase is primarily queuing — more clients waiting for their turn. +6. **Metadata-only operations degrade gracefully**: deleteItem/renameItem p50 grows from ~750ms (10 clients) to ~3.0s (200 clients). ## Upload Flow Latency Breakdown (Prometheus Server-Side) -Captured from staging `/metrics` endpoint after all load test runs (10–100 clients). Cumulative histograms. +Captured from staging `/metrics` endpoint after all load test runs (10–200 clients). Cumulative histograms. ### Where uploadFile time goes @@ -158,14 +160,15 @@ A single SDK `uploadFile` call makes 5 sequential API calls: ### Internal operation timing -| Metric | Count | Mean | Notes | -| ------------------------------- | ------- | ----- | -------------------------------------------------- | -| IPFS Pin (Kubo `pin add`) | 152,930 | 1.56s | ~95% of upload endpoint time | -| HTTP POST /ipfs/upload | 152,930 | 1.64s | Pin + quota check + DB write | -| IPNS Publish (DB upsert) | 111,300 | 134ms | Fast — just a database write | -| IPNS Publish Batch (DB upsert) | 20,786 | 98ms | Similar to single publish | -| Async DHT propagation (success) | 131,762 | 861ms | Fire-and-forget, does not block client | -| Async DHT propagation (error) | 324 | 17.2s | 0.25% error rate, does not affect client responses | +| Metric | Count | Mean | Notes | +| ------------------------------- | ------- | ----- | --------------------------------------------------- | +| IPFS Pin (Kubo `pin add`) | 172,387 | 1.56s | ~95% of upload endpoint time | +| HTTP POST /ipfs/upload | 172,387 | 1.64s | Pin + quota check + DB write | +| IPNS Publish (DB upsert) | 122,892 | 127ms | Fast — just a database write | +| IPNS Publish Batch (DB upsert) | 24,717 | 92ms | Similar to single publish | +| IPNS Publish (409 conflict) | 1 | 4ms | Expected at high concurrency (seq number collision) | +| Async DHT propagation (success) | 147,285 | 838ms | Fire-and-forget, does not block client | +| Async DHT propagation (error) | 324 | 17.2s | 0.22% error rate, does not affect client responses | ### Bottleneck analysis