Tree-sitter chunker fails on deep C++ ASTs (recursion → segfault); no per-file isolation; no CLI index persistence

[zlareb1]

First — thanks for shipping semble; the agent-first design + zero-API-key install is exactly what's missing from the code-search-for-agents space. We piloted it against a large C++ codebase (ClickHouse, ~25K files, ~2M LOC of templated C++) and hit a chain of three issues that compound. Reporting all three together because (2) and (3) are why (1) is unrecoverable rather than just slow.

Repro target

• Repo: ClickHouse/ClickHouse at master
• Version: semble==0.1.10 installed via uv tool install semble
• Python: 3.10.20 (uv-managed)
• Kernel: Linux 6.1, aarch64
• Filesystem: ext4

Issue 1 — RecursionError in chunker on deeply-templated C++ ASTs

File ".../semble/chunking/core.py", line 93, in _merge_node_inner
    groups.extend(_merge_node_inner(child, desired_length))
  [Previous line repeated 987 more times]
RecursionError: maximum recursion depth exceeded

ClickHouse has many template-heavy .cpp/.h files (MergeTreeData.cpp and friends) where the tree-sitter parse tree nests deeper than Python's default 1000-frame recursion limit. _merge_node_inner recurses unboundedly on AST depth and trips the limit.

Suggested fix: cap the recursion depth in _merge_node_inner and switch to an iterative fallback (or a coarser chunk) past the cap. Files that would have been chunked at a finer grain still get chunked, just less precisely.

Issue 2 — Bumping setrecursionlimit segfaults instead of erroring

sys.setrecursionlimit(50000) made the chunker run further but produced a coredump (no Python traceback). Python's recursion counter governs Python frames; the underlying C stack is bounded by RLIMIT_STACK (~8 MB default on Linux). At ~50k Python frames the C stack overflows and the process dies with SIGSEGV. So the apparent workaround is strictly worse than the original — it converts a recoverable Python exception into an uncontrolled process exit.

This is why Issue 1 is the only safe failure mode today, and why fixing it via cap-and-fallback (rather than "just raise the recursion limit") is the right shape.

Issue 3 — No per-file error isolation in chunk_source

Even with Issue 1 fixed for typical files, any single file that still trips the chunker aborts the entire index build. There's no try/except around chunk_source(file) in the indexing loop, so one bad file kills indexing for the other 24,999 files.

Suggested fix: wrap per-file chunking in try/except, log the file path + exception class, and skip. The resulting index is partial-but-useful instead of nonexistent. Operator gets a clear list of "these files weren't chunked" they can inspect.

Issue 4 (bonus) — CLI mode doesn't persist its index

Once we worked around Issues 1-3 to index a small subset (the SQL-only test corpus, ~8,500 files, no C++), semble search from the CLI re-indexed on every invocation. The MCP-server form caches indexes per session, but agents that invoke semble via shell subprocess (i.e., agents not using MCP — many production agent frameworks shell out for tools) pay the index cost on every call. On the test-corpus subset that's 24 seconds per call; on a larger surface where the cap-and-skip fixes above let indexing complete, it'd be much higher.

Suggested fix: persist the index to disk (e.g., <repo>/.semble/index.bin or ~/.cache/semble/<repo-hash>/index.bin) and load on subsequent CLI invocations when source mtimes haven't moved. Same semantics as the MCP-server cache, just persisted across processes.

Why this combo matters

Issues 1+3 together mean a single deeply-templated file kills indexing for the entire repo, with no fallback. Issue 4 means the operator can't even amortize the index cost by running queries in batch — every call rebuilds. For agents shelling out 3-5 times per investigation, that's 70-120 seconds per task on a small subset, more on the full repo.

Happy to test patches against ClickHouse master if you want a real-world repro, and to share the .sembleignore we used (.worktrees/, build/, the large contrib/ subtrees) so you can reproduce locally without needing a full ClickHouse checkout.

Excellent project; really hoping this gets unblocked for C++ workloads at this scale.

[stephantul]

Hello!

This is solved on main, see #122. We'll likely release today. If you're up for it, you can install main and see if it works.

We're working on a reasonable solution for 4. Persistence is a difficult issue, however. For example ~/.cache/semble/<repo-hash> doesn't reflect local changes, what if multiple agents make changes in parallel, and so on. So we'd need some kind of file watching structure etc. etc.

[zlareb1]

Thanks for the update @stephantul. Would it be possible for you to try it on major codebase like https://github.com/ClickHouse/ClickHouse before release to ensure it works correctly?

I would actually wait for release to try it out.

[stephantul]

Hey,

We just released. We tried it on a larger codebase, not clickhouse. Feel free to try it out!

[zlareb1]

@stephantul — confirming v0.2.0 fixes items 1-3 on a real ClickHouse checkout. Tested by indexing src/ (83 MB, 25K-ish C++ files):

• v0.1.10: RecursionError → setrecursionlimit workaround → C-stack segfault. Could not index at all.
• v0.2.0: completes in ~40s, logs "Recursion depth exceeded in chunk." for the deeply-templated files and skips them rather than aborting the whole index. Identifier queries (MergeTreeData::loadDataParts) return the right call site. Excellent fix; thank you.

Full repo (~25K files including contrib/, tests/, programs/) still times out at 10 min, but that's manageable on our end with a .sembleignore excluding contrib/, build/, and tests/integration/. Not asking for anything on scale — that's our problem to scope.

On item 4 (CLI index persistence) — why it matters for our use case

I want to surface the use case so it's not just an abstract feature request. We're building ClickGap, an autonomous PR review agent for ClickHouse. Architecturally:

• ClickGap agents call into shell tools via the Claude Agent SDK's Bash tool (the standard Bash,Read,Grep,Glob,Write toolkit). We don't use MCP — the agents shell out to grep / find / clickhouse-test the same way they'd shell out to semble.
• A typical gap_analyst investigation runs for 60-90 minutes / 100 turns of Opus, and during that window it makes 3-5 distinct code-search calls (looking up function definitions, finding related test patterns, locating call sites for a hypothesis).
• At 40s of cold-index tax per semble search call, that's 2-3 minutes of cumulative re-indexing per investigation, on top of the actual query latency.

The alternative the agent has today is grep -rn against src/, which runs in well under a second and is exhaustive. So the per-call math is roughly:

Approach	Per-call latency	Per-investigation overhead	Recall on conceptual queries
grep	<1s	<5s	weak (exact match only)
semble v0.2.0 CLI	~40s	~120-200s	better (semantic + identifier hybrid)
semble w/ CLI persistence	~1s expected	~5s	same as above

The middle row is what we measured today. The bottom row is the configuration where the math flips and the pilot becomes worth running for us — better recall AND comparable cost.

I understand the concern you raised about parallel agents making changes in parallel. For our use case specifically, the index is mostly read-only between agent runs (the source tree only changes when we git fetch upstream master, which we control and can trigger reindex on). A "build once, reuse until the operator says reindex" model — ~/.cache/semble/<repo-hash>/ keyed on repo HEAD SHA, or just semble index <path> + semble search --index <path> — would cover the agent-bash use case cleanly without needing file-watching. Mid-run staleness against the master copy is a feature, not a bug, for us: we want the analyst to see a stable snapshot per finding.

Different use cases will want different staleness semantics, of course — happy to share more context if it'd help your design. The headline ask is just: please consider an explicit-reindex CLI mode as the simplest variant before tackling the harder "track filesystem changes automatically across N concurrent agents" version. Thanks for the v0.2.0 release.

[stephantul]

Thanks for checking, happy this solved your issue and thanks for being so responsive 🙏 !

Ok your explanation makes a lot of sense. Then I guess something like what you proposed makes sense.

semble index "path/to/index"
semble search "query" --index "path/to/index"

If the --index argument is empty, the index is built in memory and discarded.

Then later, we could extend this to a reindexed component.

[stephantul]

We've just added this in PR: #140. It's pretty big so it may not make it through today or tomorrow, but we'll include it in the next release.

[Pringled]

Hey @zlareb1, we've released this in v0.3.0. Caching and invalidation is done automatically, let us know if this works for your usecase!