Idea: add LSP (language-server) tooling to the dev-lead / pr-review agent harness

[don-petry]

TL;DR

Adding LSP (Language Server Protocol) tooling to our agent harness is worth piloting — it gives the agents real semantic code intelligence (go-to-definition, find-references, type resolution, diagnostics) instead of grep + file-reading. The clearest wins are dev-lead writing/fixing code and pr-review verifying findings across files. The one catch unique to us: LSP servers must cold-start and index on every ephemeral CI run, so it needs a caching + tier-gating strategy. It would not have helped the config/YAML/secrets incidents we hit recently — LSP raises code-intelligence, not config-intelligence.

---

The problem it solves (in our terms)

Our agents navigate code by grep + reading files. That fails exactly where they operate:

• Cross-file evidence chains — a review or CI-fix that follows a symbol through imports/types across the repo. Grep returns textual matches (validateToken in a string/comment), not semantic ones.
• Third-party / transitive types — resolving a library signature means walking node_modules/site-packages or guessing → a documented source of hallucinated APIs (~20% of AI package suggestions point at libraries that don't exist).
• Context-window slicing — large repos don't fit in context, so the agent gets a "sliced, incomplete, often misleading view." Our batch reviewer and dev-lead fixers hit this constantly.

LSP replaces that with textDocument/definition, references, hover, diagnostics, documentSymbol, rename — exact source locations + fully-resolved types, exposed to the agent as MCP tools.

Quantified benefits

• Token efficiency, 5–34× vs grep/read on the same task. Cited blast-radius analysis on 319K-line Consul: 17.7 MB → 841 KB, 5,534 tool calls → 119. Find-references in a 100-file project ~500 tokens (LSP) vs 2,000+ (grep).
• ~20–30 min of manual file-tracing saved per error (TypeScript).
• Accuracy: a validation phase that cross-checks each finding against real codebase context is what separates "confidently wrong" from "evidence-based" review — and LSP is that validator.

Why this matters to us specifically: we run a Token Cost Observatory and just had a Gemini-credits-depleted incident. A 5–34× cut on navigation-heavy turns is a direct hit to per-run token spend and rate-limit pressure.

Where it helps in our harness

• dev-lead (writing code) — strongest win. Correct signatures on the first pass, safe rename, diagnostics to self-check before pushing → fewer broken commits.
• pr-review (reviewing code) — high value, different shape. Use LSP as a grounding/verification step in the deep/audit tiers: before posting "this is undefined / breaks N callers," confirm via lsp_get_diagnostics / lsp_find_references. Directly attacks the false-positive problem (30–42% of AI-review false positives come from misreading context) — which is exactly what erodes trust in an auto-approving agent.
• Reviewing our own Shell. Our biggest "language" is the harness itself (engine.sh, review-batch.sh, compliance-audit.sh). bash-language-server + existing shellcheck give the agents real navigation over their own substrate.

Costs / caveats specific to us

• Ephemeral runners are the catch. The vendor write-ups assume a persistent, warm-indexed server (IDE/CLI). We run per-PR, time-boxed jobs (60-min cap; triage 300 s / deep 600 s tiers). An LSP server (tsserver, gopls, jdtls) must cold-start + index on every run — minutes on a big repo, eating tier budgets. This is the make-or-break operational issue. Mitigations: actions/cache the LSP index, scope to changed files, gate LSP to deep/audit only, lazy single-file diagnostics.
• Multi-engine is per-engine. LSP reaches the agent as MCP tools. Claude (primary) and Copilot CLI (now Go-live improvements for PR review agent #2 after the fallback-order fix) support MCP/skills; Gemini CLI's MCP support is weakest — degrades on our last-resort engine.
• Multi-language management. Per-language servers to install/version on the runner. Fleet is JS/TS/Python/Shell (manageable); Google Apps Script has no first-class LSP.
• Overlap with existing tooling. We already run SonarCloud, CodeQL, shellcheck. LSP's distinct value is interactive semantic navigation during the agent's reasoning, not batch linting — complementary; scope so we don't pay twice.
• Honest limit: LSP would not have prevented this cycle's incidents (broken @v1 workflow ref, channel-tag pins, the markets PAT override) — those are YAML/config/secrets, outside LSP's wheelhouse.

Recommendation — phased, low-regret

1. Pilot dev-lead on one language/repo (TypeScript in markets, or Shell in .github-private) using an existing LSP-MCP server (e.g. Serena, or the 24-tool CI-verified lsp MCP server) — don't build one.
2. Gate LSP to deep/audit tiers and cache the index to keep cold-start off the critical path.
3. Add a pr-review verification hook that resolves findings via lsp_get_diagnostics/lsp_find_references before posting — measure false-positive rate before/after.
4. Measure with the Token Cost Observatory: tokens/run, tool-call count, review precision — pilot vs control. The 5–34× claim is testable on our own repos.
5. Go/no-go fleet-wide on the data. (MCPBench caution: "using MCPs does not always show noticeable improvement vs plain function-calls" — the measured pilot matters; don't roll out on the vendor narrative alone.)

Open questions for discussion

• Start with dev-lead (code-writing) or pr-review (finding-verification) first?
• Is the per-run indexing cost acceptable, or do we need a persistent/warm LSP service (e.g. a sidecar) before this is viable on CI?
• Which pilot language/repo gives the cleanest signal — TS in markets or Shell in .github-private?

---

Sources

• github.blog — Copilot CLI + language servers
• MindStudio — LSP + Claude Code on large codebases
• Serena MCP server · LSP MCP (24-tool, CI-verified)
• MCPBench evaluation (arXiv) · CR-Bench: AI code-review utility (arXiv)
• Agentic AI code review: confidently-wrong → evidence-based · Diagnosing AI-review false positives

_{🤖 Researched & drafted with Claude Code}

[github-actions[bot]]

Weekly Update — Deep Research: LSP-MCP Server Selection, Caching Design & Tier Gating

What Changed

Major new market signals since initial writeup:

1. GitHub Copilot CLI shipped an LSP Setup skill (June 10, 2026) — the platform vendor now considers LSP-backed code intelligence table stakes for agentic coding. This validates the direction and adds urgency: if we don't add LSP, our agents will be behind the Copilot CLI baseline.

2. agent-lsp (Blackwell Systems) emerged as a strong alternative to Serena and lsp-mcp:

   • Single Go binary, 65 tools + 24 skills, 30 CI-verified languages
   • Persistent daemon with ~10s first-index warm-up, auto-exits after 30min
   • Speculative execution: preview_edit simulates changes in memory before writing — the agent sees what breaks before touching disk
   • GCF (Graph Compact Format): 30–92% token reduction vs JSON responses (92.7% savings by the 5th call with session deduplication)
   • --non-interactive init and --http mode support CI scripting
   • github.com/blackwell-systems/agent-lsp | blog.blackwell-systems.com

3. ManoMano's Project AEGIS benchmark (Feb 2026, 36K-line Java codebase):

   • Vanilla Claude failed a complex refactoring task in 60 min
   • Claude Code + built-in LSP: build failed, 9/1017 tests broken
   • Claude + Serena MCP: 45 min, all 1,017 tests passing, $27.30 (vs $23.54 for vanilla Claude that failed)
   • Key insight: LSP's value is not cheaper per-token — it's avoiding expensive failures
   • ManoMano Tech — Project AEGIS

4. TypeScript 7.0 beta (April 2026) ships tsgo — a Go-native LSP server that is 10x faster than tsserver. Editor startup drops from 9.6s → 1.2s. This largely eliminates the cold-start concern for TypeScript projects on ephemeral runners.

5. MCP token overhead quantified (MindStudio 2026): each MCP tool definition costs ~200 tokens/turn. With 65 tools, that's ~13,000 tokens overhead per turn. Mitigation: expose only 8-12 navigation tools → ~2,000 tokens overhead. Source

6. Gemini CLI status: MCP support exists but is the weakest of the three engines. Unpaid tier transitions to Antigravity CLI on June 18. Our fallback engine should degrade gracefully — skip MCP tools entirely when running on Gemini/Copilot fallback.

Research Answers to Open Questions

Q1: Start with dev-lead (code-writing) or pr-review (finding-verification) first?

Recommendation: pr-review deep tier first. Rationale:

• pr-review runs on every PR — higher signal volume for pilot data
• The finding-verification use case (confirm via find_references/get_diagnostics before posting) is lower risk than code-writing: a bad verification downgrades a finding, a bad code write breaks a build
• The Token Cost Observatory already captures per-PR metrics — pilot vs control comparison is immediate
• dev-lead follows once pr-review pilot validates the indexing cost on real workloads

Q2: Per-run indexing cost — acceptable or need persistent/warm LSP service?

Acceptable with mitigations, no sidecar needed. Key data points:

• agent-lsp: ~10s first-index for typical projects, instant on warm daemon
• tsgo: 1.2s cold start (TS projects are essentially free)
• bash-language-server: lightweight, Tree Sitter based, instant for shell-only repos
• gopls v0.12+: incremental indexing, 75% memory reduction
• 10s indexing on a 600s deep-tier budget = 1.7% overhead — easily acceptable
• Caching strategy: actions/cache keyed on ${{ runner.os }}-lsp-${{ hashFiles('**/package-lock.json', '**/go.sum') }}-${{ env.LSP_SERVER_VERSION }}, matching the existing Claude CLI binary cache pattern (pr-review.yml:197-201)

Q3: Pilot language/repo — TS in markets or Shell in .github-private?

Start with Shell in .github-private (this repo). Rationale:

• This repo is the agent's own substrate — it reviews its own scripts (engine.sh, review-batch.sh, etc.)
• bash-language-server is lightweight and fast — minimal indexing overhead
• Self-hosting the pilot means faster iteration (no cross-repo coordination)
• The review pipeline for this repo already exercises all tiers (triage→deep→duck→audit)
• Follow up with TypeScript (markets) once Shell pilot validates the architecture

Q4: Which LSP-MCP server — Serena vs lsp-mcp vs agent-lsp?

Recommendation: agent-lsp. Comparative analysis:

Criterion	Serena	lsp-mcp	agent-lsp
Language	Python (uv)	Go	Go
CI verification	Not documented	7 languages	30 languages, every push
Cold start	Not documented	Not documented	~10s, daemon persists
Token optimization	N/A	JSON only	GCF: 30-92% reduction
Speculative execution	No	No	Yes (preview_edit, simulate_chain)
Tool count	~15	~24	65 tools + 24 skills
Binary distribution	Python wheel (uv)	Go binary	Go binary
CI scripting	Not recommended	Possible	--non-interactive, --http
Benchmark data	AEGIS ($27.30, 45min, success)	None	None

Serena has the production benchmark (AEGIS), but agent-lsp's CI-first design (Go binary, non-interactive mode, GCF token compression, 30-language CI matrix) better fits our ephemeral-runner, multi-engine architecture. Pilot agent-lsp; use Serena results as the success bar to beat.

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	high	high	→
Impact	high	high	→ (confirmed by AEGIS benchmark + Copilot CLI LSP skill)
Urgency	med	high	↑ (Copilot CLI shipped LSP June 10; platform is moving)

Recommendation

Advance — start the pilot. All five open questions from the original writeup now have data-backed answers. The integration path is clear: add --mcp-config to run_agentic() in engine.sh for deep/audit tiers only, install agent-lsp Go binary in runner setup, cache the index via actions/cache. Three companion proposals filed as separate Ideas discussions:

1. 💡 LSP-Grounded Finding Verification — the specific pr-review false-positive reduction mechanism
2. 💡 Per-Tier MCP Tool Gating — the engine.sh architecture for selective tool exposure
3. 💡 Token Cost Observatory LSP Metrics — the measurement framework for go/no-go

Sources

• GitHub Blog — Copilot CLI + Language Servers (June 10, 2026)
• agent-lsp (Blackwell Systems) | Blog post
• ManoMano — Project AEGIS Benchmark
• TypeScript 7.0 Beta — tsgo 10x faster
• MindStudio — MCP Token Overhead
• arXiv:2604.10800 — Verify Before You Fix: Execution Grounding
• SonarQube Agentic Analysis MCP Server
• Serena MCP Server
• Gemini CLI MCP Server docs

_{🤖 Researched & drafted by Mary (BMAD Strategic Business Analyst) with Claude Code}

[github-actions[bot]]

Enhancement: LSP Integration — Cold-Start Budget, Caching Strategy & Targeted Use Cases

Sharpened problem & goal
Adding LSP tooling gives agents semantic code intelligence (go-to-definition, find-references, type resolution) instead of grep + file-reading. The idea correctly identifies the core challenge: LSP servers must cold-start and index on every ephemeral CI runner. The critical decision before piloting: which specific agent actions benefit most from semantic intelligence, and what is the acceptable cold-start latency budget? Without this scoping, the pilot may instrument an LSP server for use cases where grep is already sufficient.

Context
Relevant existing infrastructure:

• scripts/dev-lead-preflight.sh — pre-flight checks before dev-lead engages; LSP warm-up is a natural preflight step, keeping the cost visible and skippable
• scripts/engine.sh — the agentic execution layer; an LSP-aware tool wrapper would be registered here
• The idea correctly notes LSP would not have helped recent config/YAML/secrets incidents — keep the pilot scoped to code-intelligence tasks (cross-file refactors, import resolution, type-check violations) where semantic tools beat grep

Caching strategy for cold-start: GitHub Actions actions/cache can persist LSP index files (e.g., rust-analyzer's target/, clangd's .cache/, Pyright's type stubs) keyed by repo + language + lock file hash (Cargo.lock, go.sum, poetry.lock). This is already standard for build caches and reduces cold-start to a cache-hit restoration, not a full re-index.

Token economics: LSP reduces tool-call tokens (fewer grep + multi-file reads needed to resolve a reference) but adds runner time. The Token Cost Observatory (token-report.yml) should measure the tradeoff.

Impact / Effort

• Impact: MEDIUM — high for repos with complex cross-file references (large TypeScript, Rust); low for repos that are primarily shell/YAML where grep is already accurate; the pilot repo choice is critical
• Effort: L — selecting + integrating one LSP server with cache warming in dev-lead-reusable.yml is M; generalization to multiple languages and tier-gating is L

Suggested acceptance criteria

• A pilot scoping doc (docs/lsp-pilot.md) defines: target language (one to start), specific dev-lead actions that use it, and the acceptable cold-start SLA (e.g., ≤30s P95)
• LSP server warm-up is a preflight step in scripts/dev-lead-preflight.sh with actions/cache keyed on the language ecosystem lock file hash — gracefully skipped (not a failure) if cache miss exceeds SLA
• Cold-start time is measured and reported in the workflow step summary (P50/P95 over 2 weeks) against the scoped SLA
• Token usage before/after is compared via token-report.yml — LSP should reduce tool-call tokens for cross-file lookups; the pilot report must include this comparison
• If cold-start exceeds SLA for a given repo, the LSP step is automatically skipped — graceful degradation, never a workflow failure

[don-petry]

Agreed, and, we should perform a set of comparative tests that measure the speed, quality and cost of a variety of PR's with and without the LSP. If there is strong competition between two LSP options, then we can use the comparative tests to determine our selection.

[github-actions[bot]]

📋 Initiative planned by the BMAD Scrum Master (Bob).

Epic #839 — Initiative: Pilot LSP (language-server) code intelligence in the agent harness — pr-review Shell first, measured go/no-go

7 stories created (inert — labelled initiative, NOT initiative:auto):

• [Phase 1] Author the LSP pilot scoping doc — language, actions, SLA, metrics, cost cap, candidate shortlist #840 (S) — [Phase 1] Author the LSP pilot scoping doc — language, actions, SLA, metrics, cost cap, candidate shortlist
• [Phase 1] Freeze the comparative pilot PR corpus + immutable LSP-off baseline and build the metric-extraction harness #841 (M) — [Phase 1] Freeze the comparative pilot PR corpus + immutable LSP-off baseline and build the metric-extraction harness
• [Phase 2] Wire candidate LSP-MCP server(s) into the pr-review runtime — config, install, tier gating #842 (M) — [Phase 2] Wire candidate LSP-MCP server(s) into the pr-review runtime — config, install, tier gating
• [Phase 2] Add the LSP finding-verification step to the deep/audit pr-review tiers #843 (M) — [Phase 2] Add the LSP finding-verification step to the deep/audit pr-review tiers
• [Phase 3] Run the comparative pilot and render the speed/quality/cost report #844 (M) — [Phase 3] Run the comparative pilot and render the speed/quality/cost report
• [Phase 3] Record the human go/no-go decision and fleet-wide recommendation #845 (S) — [Phase 3] Record the human go/no-go decision and fleet-wide recommendation
• [Phase 2] Cache the LSP index and enforce the 30s cold-start SLA with auto-skip #846 (M) — [Phase 2] Cache the LSP index and enforce the 30s cold-start SLA with auto-skip

Open questions for review:

• Should the pilot corpus include TypeScript PRs from markets for a cross-language signal, or is Shell-only in .github-private sufficient for the go/no-go? The research recommends Shell-first with TS as a follow-up; this epic is scoped Shell-only unless the owner wants TS folded in.
• Beyond the structural run-count cost cap (<=120 deep-tier runs) and the 30s P95 cold-start auto-skip, does the owner want an explicit dollar/token ceiling per pilot run recorded in docs/lsp-pilot.md?

---

Review the epic and its sub-issue DAG, adjust as needed, then add initiative:auto to epic #839 to hand it to initiative-driver for auto-implementation.

[don-petry]

Pilot resolved — NO-GO for Shell/bash-language-server; door left open per-language

The Shell pilot (epic #839) is complete. On the real 20-PR corpus, LSP-on delivered no navigation-token reduction — sonnet-4-6 1.26× / opus-4-8 (prod deep tier) 1.05× aggregate, high-nav stratum a wash at ~1.0×, vs the ≥2× target. Cold-start passed. Decision: NO-GO, recorded on #845 (docs/lsp-pilot.md §7).

The result is scoped to Shell/bash-language-server and doesn't generalize. Follow-up tracking for the languages we actually use is filed as #1124 (umbrella, inert) → #1125 Python/pyright, #1126 TypeScript-JS. The measurement harness stays in-repo, dormant, for a low-friction per-language restart.