Skip to content

feat: item 8 (search-mcp cartridge) + ADR-0013 streamable HTTP transport (epic #87)#99

Merged
hyperpolymath merged 1 commit into
mainfrom
feat/item-8-search-mcp
May 20, 2026
Merged

feat: item 8 (search-mcp cartridge) + ADR-0013 streamable HTTP transport (epic #87)#99
hyperpolymath merged 1 commit into
mainfrom
feat/item-8-search-mcp

Conversation

@hyperpolymath

Copy link
Copy Markdown
Owner

Summary

Two epic #87 Tier A items in one PR — item 8 ships as code, item 14 ships as an RFC (the impl is ~1-2 weeks of focused work per epic estimate; design-first per the Tier B pattern).

Item 8 — `search-mcp` cartridge + `boj_search` bridge tool

Multi-provider web search behind one cartridge:

Provider Strength Auth env
Tavily LLM-optimised summaries; answer-mode; extract API `TAVILY_API_KEY`
Brave Privacy-first index; no tracking `BRAVE_SEARCH_API_KEY`
Exa Neural search over high-quality content; extract API `EXA_API_KEY`
Perplexity Q&A-style with citations `PERPLEXITY_API_KEY`

What's in the diff

  • `cartridges/search-mcp/cartridge.json` — manifest with 4 cartridge-level tools (`search_authenticate` / `web` / `answer` / `extract`)
  • `cartridges/search-mcp/README.adoc` — provider strengths matrix + bridge wiring note
  • `mcp-bridge/lib/tools.js` — single `boj_search` bridge tool dispatching via `operation` + `provider` args; AAA-tier description carries the provider matrix so the LLM picks correctly
  • `mcp-bridge/main.js` — dispatch case + hardeningGate requires `operation` + domain classification `search`
  • `mcp-bridge/lib/offline-menu.js` — `search-mcp` added to Teranga tier so it shows in `boj_menu` even offline

Status

Bridge surface captured — MCP clients discover and invoke today. Backend dispatch implementation in Elixir/Zig deferred (separate work).

Tests

  • ✅ 15/15 existing tests pass
  • ✅ `tools/list` advertises `boj_search` with full 41+1 = 42 tools

Item 14 — ADR-0013 Streamable HTTP transport

RFC for adding MCP's HTTP+SSE transport alongside stdio. Same bridge binary; `BOJ_TRANSPORT=stdio|http|both` selects mode. No existing stdio clients break.

Why this RFC matters

JSR's runtime-compatibility question surfaced the gap: the user can't tick "Browsers" or "Cloudflare Workers" because the bridge legitimately doesn't run there. Item 14 is the right framing — not "make stdio work in a browser" (architecturally wrong) but "add an HTTP transport for the contexts where stdio doesn't work."

Key design points

  • Single endpoint `POST /mcp` + `GET /mcp` (SSE) per latest MCP spec
  • Session state via `Mcp-Session-Id` UUIDs (rate-limit counters, OTel trace context, in-flight async work)
  • Four auth modes: none (loopback-only), bearer, mTLS, OIDC
  • Mounts on the same Cowboy listener as webhooks (ADR-0004 honoured)
  • Cartridge compat matrix: ~60% work on Workers (HTTP-API-based: GitHub, GitLab, Cloudflare, ML, Search, Hugging Face); 40% don't (local-only: browser-mcp, container-mcp, local-coord-mcp, local sandbox). New `boj_capabilities` resource reports per-deployment availability so clients don't try locally-impossible operations.

Three new deployment targets unlocked

  • Cloudflare Workers (with Durable Objects for session state)
  • Docker / Podman containers
  • Browser-based agent web apps (via `fetch` against a hosted BoJ)

Implementation sketch

2 PRs over ~1-2 weeks:

  • PR 1: bridge HTTP transport + auth modes + session manager (~1 week)
  • PR 2: Workers deployment guide + Durable Objects shim + cold-start optimisation (~1 week)

Federation alignment

ADR-0010 (cross-machine federation) needs HTTPS transport anyway. ADR-0013's transport layer is reusable for federation. Webhook notifications (ADR-0011) fan out over this RFC's SSE stream.

Sequencing

Independent of all currently-open PRs. Item 8 lands as code; item 14 lands as RFC awaiting implementation in follow-up. After this lands, epic #87 Tier A remaining is items 7 (vector DBs × 4) + 9 (multi-modal × 4) — both multi-cartridge waves that warrant separate sessions, not a single rushed PR.

🤖 Generated with Claude Code

docs(item-14): ADR-0013 Streamable HTTP transport

Two epic #87 Tier A items in one PR — search-mcp ships as code,
HTTP-transport ships as an RFC (~1-2 weeks of impl work per epic
estimate; design-first per Tier B pattern).

== Item 8 — search-mcp cartridge ==

Multi-provider web search behind one cartridge, four providers:
- Tavily — LLM-optimised summaries; answer-mode; extract API
- Brave — privacy-first index; no tracking
- Exa — neural search over high-quality content; extract API
- Perplexity — Q&A-style answers with citations

Surface:
- cartridges/search-mcp/cartridge.json — manifest with 4 cartridge-
  level tools (search_authenticate / web / answer / extract)
- cartridges/search-mcp/README.adoc — provider strengths + auth env
  vars + bridge wiring note
- mcp-bridge/lib/tools.js — single boj_search bridge tool dispatching
  to cartridge via operation + provider args (open=tavily/brave/exa/
  perplexity for web; perplexity/tavily for answer; tavily/exa for
  extract). AAA-tier description carries the provider matrix so the
  LLM picks correctly.
- mcp-bridge/main.js — dispatch case routes to search-mcp via
  invokeCartridge; hardeningGate requires operation field; domain
  classification: search
- mcp-bridge/lib/offline-menu.js — search-mcp added to Teranga tier

Backend implementation in the Elixir/Zig dispatch layer is deferred;
surface area is captured so MCP clients can already discover and
invoke (returns hint until backend lands). Existing 15/15 tests pass;
boj_search appears in tools/list.

== Item 14 — ADR-0013 Streamable HTTP transport ==

RFC for adding MCP's HTTP+SSE transport alongside stdio. Same bridge
binary; BOJ_TRANSPORT=stdio|http|both selects mode. No existing
stdio clients break.

Key design points:
- Single endpoint POST /mcp + GET /mcp (SSE) per latest MCP spec
- Mcp-Session-Id UUIDs maintained server-side for per-session state
- Auth: none (loopback-only) | bearer | mTLS | OIDC
- Mounts on the same Cowboy listener as webhooks (ADR-0004 honoured)
- Workers deployment: ~60% of cartridges work (HTTP-API-based);
  local-only cartridges (browser-mcp, container-mcp, local-coord-mcp,
  local sandbox) don't and are reported via boj_capabilities
- Federation (ADR-0010) reuses this transport layer
- Webhook notifications (ADR-0011) fan out over the SSE stream

Three new deployment targets unlocked: Workers, containers,
browser-based agents.

Implementation plan: 2 PRs over ~1-2 weeks. Tracked in epic #87 item 14.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🔍 Hypatia Security Scan

Findings: 33 issues detected

Severity Count
🔴 Critical 19
🟠 High 5
🟡 Medium 9

⚠️ Action Required: Critical security issues found!

View findings
[
  {
    "reason": "Stale AI session file -- delete",
    "type": "stale",
    "file": "GEMINI.md",
    "action": "delete",
    "rule_module": "root_hygiene",
    "severity": "medium"
  },
  {
    "reason": "Issue in quality.yml",
    "type": "missing_workflow",
    "file": "quality.yml",
    "action": "create",
    "rule_module": "workflow_audit",
    "severity": "high"
  },
  {
    "reason": "Issue in security-policy.yml",
    "type": "missing_workflow",
    "file": "security-policy.yml",
    "action": "create",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Action actions/cache@v4 needs attention",
    "type": "unpinned_action",
    "file": "abi-drift.yml",
    "action": "pin_sha",
    "rule_module": "workflow_audit",
    "severity": "medium"
  },
  {
    "reason": "Action hyperpolymath/standards/.github/workflows/governance-reusable.yml@main needs attention",
    "type": "unpinned_action",
    "file": "governance.yml",
    "action": "pin_sha",
    "rule_module": "workflow_audit",
    "severity": "high"
  },
  {
    "reason": "Python file detected -- banned language",
    "type": "banned_language_file",
    "file": "/home/runner/work/boj-server/boj-server/.github/scripts/validate-eclexiaiser.py",
    "action": "flag",
    "rule_module": "cicd_rules",
    "severity": "critical"
  },
  {
    "reason": "TypeScript file detected -- banned language",
    "type": "banned_language_file",
    "file": "/home/runner/work/boj-server/boj-server/cartridges/sanctify-mcp/adapter/mod.ts",
    "action": "flag",
    "rule_module": "cicd_rules",
    "severity": "critical"
  },
  {
    "reason": "TypeScript file detected -- banned language",
    "type": "banned_language_file",
    "file": "/home/runner/work/boj-server/boj-server/cartridges/academic-workflow-mcp/adapter/mod.ts",
    "action": "flag",
    "rule_module": "cicd_rules",
    "severity": "critical"
  },
  {
    "reason": "TypeScript file detected -- banned language",
    "type": "banned_language_file",
    "file": "/home/runner/work/boj-server/boj-server/cartridges/fireflag-mcp/adapter/mod.ts",
    "action": "flag",
    "rule_module": "cicd_rules",
    "severity": "critical"
  },
  {
    "reason": "TypeScript file detected -- banned language",
    "type": "banned_language_file",
    "file": "/home/runner/work/boj-server/boj-server/cartridges/ephapax-mcp/adapter/mod.ts",
    "action": "flag",
    "rule_module": "cicd_rules",
    "severity": "critical"
  }
]

Powered by Hypatia Neurosymbolic CI/CD Intelligence

@hyperpolymath hyperpolymath merged commit 6d6e629 into main May 20, 2026
15 of 23 checks passed
@hyperpolymath hyperpolymath deleted the feat/item-8-search-mcp branch May 20, 2026 07:30
hyperpolymath added a commit that referenced this pull request May 20, 2026
…ultimodal (#103)

## Summary

Closes the bridge-surface portion of epic #87 items 7 and 9. Backend
dispatch (Elixir/Zig) tracked separately in tracker issues #100 (vector)
and #101 (multi-modal).

Eight new cartridges in two domains; two new umbrella bridge tools.

## Item 7 — Vector databases (4 cartridges)

| Cartridge | Provider strength |
|---|---|
| \`pinecone-mcp\` | Hosted serverless indexes; dimension + metric
required at create-time |
| \`weaviate-mcp\` | Hybrid (vector + BM25) search; schema-driven
classes; modular vectorisers |
| \`qdrant-mcp\` | Rust-native; payload filtering; sparse + dense
vectors; self-host or cloud |
| \`chromadb-mcp\` | Embedded or client/server; LLM-app-focused;
document storage alongside vectors |

All four share a **7-operation manifest**: \`authenticate\` /
\`list_collections\` / \`create_collection\` / \`delete_collection\` /
\`upsert\` / \`query\` / \`delete\`. Provider-specific extras live under
\`params\` (Pinecone namespace, Weaviate alpha-hybrid weight, Qdrant
score_threshold, Chroma embedding_function).

## Item 9 — Multi-modal (4 cartridges)

| Cartridge | Role | Worker-compat? |
|---|---|---|
| \`whisper-mcp\` | Speech-to-text (OpenAI API + local whisper.cpp
backends) | ✅ HTTP API |
| \`elevenlabs-mcp\` | Text-to-speech, voice cloning (premium),
multilingual | ✅ HTTP API |
| \`replicate-mcp\` | Image/video/audio generation; async prediction
model | ✅ HTTP API |
| \`ffmpeg-mcp\` | Local transcoding (probe/extract/concat/trim) | ❌
**Local-only** per ADR-0013 |

ffmpeg-mcp is the glue: extract audio from video → whisper transcribe;
extract frames → replicate vision model; etc.

## Bridge tools (2 new)

Both follow the \`boj_search\` umbrella pattern from PR #99:

\`\`\`
boj_vector {operation, provider ∈ {pinecone|weaviate|qdrant|chromadb},
...}
boj_multimodal {operation, provider ∈
{whisper|elevenlabs|replicate|ffmpeg}, ...}
\`\`\`

Each dispatches via \`invokeCartridge\` to the right cartridge.
\`hardeningGate\` requires \`operation\`. Domain classifications added:
\`vector\`, \`multimodal\`.

## Offline-menu

All 8 cartridges added to \`mcp-bridge/lib/offline-menu.js\` (Teranga
tier) so they surface in \`boj_menu\` even when the REST backend is
offline.

## Tool count

42 (after \`boj_search\`) → **44** (after \`boj_vector\` +
\`boj_multimodal\`). Per the umbrella-tool design choice in #100/#101,
adding 8 cartridges added only 2 bridge tools — not 8.

## Tests

- ✅ 15/15 existing tests pass (no regressions)
- ✅ \`tools/list\` advertises both new tools end-to-end (verified via
\`deno run\`)

## What's NOT in this PR

- Idris2 ABI, Zig FFI, Deno adapter for any of the 8 cartridges — just
the manifest + bridge wiring
- Tests against live provider APIs — surface captured; contract testing
comes with backend impl
- Backend dispatch in the Elixir/Zig layer — per-cartridge follow-up
work (~1-2 days each per the trackers)

## Sequencing

This closes the **mechanical** portion of items 7 and 9. The trackers
(#100, #101) update on merge — backends remain open for follow-up
sessions.

Epic #87 status after this lands:
- Tier A: items 1, 8, 13 done in code; items 7, 9 **surfaces done**
(backends pending); item 14 RFC only
- Tier B: 6/6 RFCs done
- Tier C: items 11, 12 untouched

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hyperpolymath added a commit that referenced this pull request Jun 9, 2026
)

## Summary

Lands `config/gateway-policy-boj.yaml` — the **live** Verb Governance
Spec
the HCG tier-2 gateway loads via `POLICY_PATH` in staging (§2.1) and
production (§3.1) per the rollout runbook. The Phase A worked example
(`config/gateway-policy-boj-example.yaml`) is retained as the
documentation
artefact; the live file is now the operational one. Closes the
example→live
promotion item on the Phase E §1.5 checklist.

Single-lane HCG tier-2 channel (`standards#91`). Phase A (#96), B (#97),
C (#98), D (#99) are joint-closed; Phase E (`standards#100`) is the
active
phase, with multiple artefacts gating closure (§6.4 Trustfile flip is
the
last). This PR lands one tractable artefact; staging soak (§2),
production
traffic split (§3) and the §6.4 flip remain owner-driven.

## What this PR lands

- **`config/gateway-policy-boj.yaml`** — live policy file.
Content-identical
to `gateway-policy-boj-example.yaml` at promotion time. Header rewritten
  to reflect its live-file role (operational artefact, not pedagogical),
with `DEFAULT-DENY INVARIANT` reframed from "Phase A check" to
"permanent
invariant — must hold for every future gateway release". DSL v1
conformance
preserved; all 28 routes (`global_verbs: [GET, POST]`; per-route
`verbs`,
  `exposure`, `name`, `narrative`; `stealth_profile` on internal routes;
top-level `stealth: { enabled: true, status_code: 404 }`) carried
forward
  unchanged.
- **Runbook §1.5** — flips the trailing "still to be promoted from this
example before §3.1" note (on the existing `[x]` example-in-place line)
  to a discrete `[x]` item recording the live file's existence and the
divergence policy ("future BoJ-surface evolution lands in the live file;
  the example remains as the worked-example artefact").
- **Runbook §2.1 step 2** — switches staging `POLICY_PATH` from the
example
to the live file so staging exercises the same artefact that production
  will. Production §3.1 (which inherits §2.1's environment with the
  traffic-shift mechanism overlaid) needs no change.
- **Runbook header** — version 0.2 → 0.3; status line updated to
acknowledge
  the live-policy promotion.

## What this PR deliberately does NOT do

- **Close `standards#100`.** Per runbook §6.5 the joint-close happens
after
  the §6.4 Trustfile flip (`tier_2_gateway.status: PENDING → DEPLOYED`),
which itself follows the §3.3 100% production-soak window. Using `Refs`
  not `Closes` to match the established Phase E pattern (PRs #38, and
Phase D PRs #14, #22, #26, #30 — all `Refs`'d their phase issue and the
  owner joint-closed the issue once the final artefact landed). This
  deliberately diverges from the dispatch brief's literal "Closes
  hyperpolymath/standards#<phase-issue-number>" line in favour of the
canonical runbook §6.5 close-out discipline that the brief itself points
  to as the source of truth ("using the canonical sources"). The owner
  remains the sole closer of `standards#100`.
- **Touch the HCG deploy spec.** `container/gateway-deploy.k9.ncl` in
`hyperpolymath/http-capability-gateway` (PR #38) reads `POLICY_PATH` at
deploy time from the env, so the live-file cut-over is a runbook +
config
artefact change on the BoJ side, not a deploy-spec change on the gateway
  side. No companion PR on the gateway repo.
- **Diverge the live file from the example.** At promotion the two files
are content-identical. Future divergence is intentional and the live
file
  is authoritative; the example may be intentionally simpler.
- **Trigger any deploy.** No traffic shift, no staging cut-over, no §6.4
  flip happens at merge time. This is a static artefact landing.
- **Update the deploy spec's `POLICY_PATH` default.** The deploy spec
carries env-var declarations; the live-file path is operator-supplied at
  deploy time.

## Verification

- [x] DSL v1 conformance: `dsl_version: "1"`; `governance.global_verbs`
is
      `[GET, POST]`; every route has a non-empty `verbs`; `exposure ∈
      {public, authenticated, internal}`; `stealth.enabled` boolean,
      `stealth.status_code: 404` in 100..599.
- [x] All 28 example routes preserved unchanged in the live file (route
      count, `name`s, paths, verbs, exposures, narratives).
- [x] SPDX header `MPL-2.0` matches repo convention (config/, docs/).
- [x] Runbook §1.5 and §2.1 cross-references to
`gateway-policy-boj.yaml`
      and `gateway-policy-boj-example.yaml` resolve.
- [ ] Manual: `mix gateway.validate config/gateway-policy-boj.yaml`
      (gateway-side; can be run by the operator before §2.1 stand-up —
      see runbook §1.5 last open item, smoke-test).

## Channel position

```
standards#91 (parent, open)
├── #96 Phase A — closed (boj-server: contract + policy-authoring + example; gateway: -)
├── #97 Phase B — closed (gateway#10: mTLS primary path)
├── #98 Phase C — closed (gateway#11: strip; boj-server#106: TrustPolicy clause)
├── #99 Phase D — closed (boj-server#168 on 2026-06-01; gateway#12/#14/#22/#26/#30)
└── #100 Phase E — IN PROGRESS
     ├── E5 runbook draft — boj-server#128 (landed; rehearsal pending)
     ├── E1 loopback prereqs — boj-server#130/#131/#132/#165/#173 (landed)
     ├── E1 deploy spec — http-capability-gateway#38 (landed)
     ├── E1 live policy promotion — THIS PR (in review)
     ├── E1 .ctp signing — owner follow-up
     ├── E2 staging cut-over — owner follow-up
     ├── E3 telemetry verification — owner follow-up
     ├── E4 production rollout — owner follow-up
     └── §6.4 Trustfile flip + §6.5 joint-close — owner-only
```

Refs hyperpolymath/standards#91
Refs hyperpolymath/standards#100

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---
_Generated by [Claude
Code](https://claude.ai/code/session_012FiVM8R8FWBgBsUGpnXTZM)_

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant