Indexed

Background-indexed full-text search service for a local workspace target, aimed primarily at AI coding agents that need millisecond-class code search across a git repository, a standalone directory tree, or an explicit multi-directory workspace.

Project goals

• Warm, millisecond-class code search for workspaces you search repeatedly.
• Agent-friendly JSON and HTTP API on localhost (stable DTOs, explicit freshness).
• Configurable update policy: live background updates by default, or manual-only refresh through explicit rescans.
• Low operational friction: auto-start on first use, auto-exit on idle, self-healing rebuilds.

Non-goals

• Replacing rg for one-off searches or ad-hoc “search everything including vendored/generated blobs” workflows.
• Code navigation (xref, “go to definition”, semantic symbol search).
• Cross-target search across unrelated daemon targets in a single query.

Tech stack

• C# / .NET 10 (net10.0-windows, nullable enabled, preview language features).
• SQLite + FTS5 via Microsoft.Data.Sqlite (code index uses the trigram tokenizer).
• Incremental indexing via FileSystemWatcher + optional git HEAD polling + reconciliation in live mode.
• HTTP/JSON daemon on 127.0.0.1 (service) with a thin CLI client (idx).
• Serialization via System.Text.Json source generation (IndexedJsonContext).

Feature set (current)

• Literal search over indexed code with ripgrep-style output (or JSON).
• Regex search with trigram-based narrowing + .NET Regex verification.
• Prose search over extracted XML docs, comment blocks, Markdown, and plain-text files.
• Truthful auto mode that merges code and prose results, preferring prose on exact same-line collisions.
• Path filtering via gitignore-style globs: --glob and --exclude.
• Index-time include/exclude filtering (--include-index, --exclude-index) plus curated default excludes for lockfiles/minified/generated outputs.
• Configurable file-size cap (--max-indexable-file-mb / --max-indexable-file-bytes) with status-visible non-indexable-file telemetry.
• Manual update mode (--index-updates manual) for static or externally-controlled targets where only startup scans and explicit idx rescan should change the index.
• Directory-tree and directory-set targets via repeated --root flags, including continuous background indexing outside git.
• Context lines (-A, -B, -C) without re-running a full scan per query.
• Kind-aware text output for prose hits plus stable JSON contracts for agents.
• Explicit freshness (indexedRevisionToken, currentRevisionToken, pendingFileCount, isStale) for agents and scripts.
• Daemon discovery by target with idx daemons plus target-aware daemon.json metadata.
• Crash-safe persistence (SQLite WAL mode) + background compaction (bounded FTS5 merges).

Known limitations

• Windows-only today (net10.0-windows).
• Huge files are whole-file indexed today: files larger than the configured cap are treated as non-indexable. The default cap is 50 MiB; raising it can increase memory use because current indexing still reads each accepted file as one byte array before tokenization.
• Multiline regex is not supported (matches are line-oriented).
• Prose query syntax is FTS5 MATCH syntax. --mode prose does not interpret pattern as a .NET regex; in --mode auto, --regex therefore runs the code side only.
• Directory-set queries use a logical-path namespace (label/relative/path) that is stable once chosen; relabeling creates a distinct target.
• Index footprint can be large for trigram indexing; see the size-reduction docs in docs/.

Status

Stages 0 through 5 are implemented and tested. Indexed now ships code search, prose extraction, truthful auto mode, background incremental indexing, workspace-target support, and productionization hardening in one coherent implementation.

Stage	Description	Status
S0	Workspace scaffolding	Complete
S1	Enumeration + CLI + daemon bootstrap	Complete
S2	FTS5 code index + query planner	Complete
S3	Prose index + content extraction	Complete
S4	Incremental indexer (FSW, HEAD polling, reconciliation)	Complete
S5	Productionization hardening	Complete

Layout

.
├── Indexed.sln
├── Directory.Build.props
├── global.json
├── README.md
├── docs/
│   ├── Indexed-Architecture.md
│   ├── Indexed-FAQ.md
│   ├── Indexed-Workspace-Targets-Proposal.md
│   ├── Indexed-Tutorial.md
│   ├── Indexed-Usage-Guide.md
│   ├── Indexed-Architecture-Proposal.md
│   ├── Indexed-Implementation-Plan.md
│   ├── Indexed-Index-Size-Reduction-Strategies.md
│   ├── Indexed-Size-Reduction-SafeNearTerm-Plan.md
│   └── Indexed-Stage4-Incremental-Indexer-Plan.md
├── src/
│   ├── Indexed.Abstractions/    DTOs: SearchRequest, SearchResponse, Freshness, Match, etc.
│   ├── Indexed.Targets/         target identity, directory targets, logical-path rules
│   ├── Indexed.Git/             git.exe wrapper: process runner, repository operations
│   ├── Indexed.Extractors/      Roslyn + regex-based prose extraction pipeline
│   ├── Indexed.Core/            SQLite+FTS5 index, query planner, full/incremental indexers
│   ├── Indexed.Service/         HTTP daemon host, idle-exit, lifecycle management
│   └── Indexed.Cli/             CLI client (output: idx): argument parsing, daemon launcher
└── tests/
    ├── Indexed.Abstractions.Tests/
    ├── Indexed.Extractors.Tests/
    ├── Indexed.Git.Tests/
    ├── Indexed.Core.Tests/
    ├── Indexed.Service.Tests/
    └── Indexed.Cli.Tests/

Quick start

# From the repository root, build and test
dotnet build Indexed.sln
dotnet test Indexed.sln

# Run a search (CLI auto-starts the daemon)
dotnet run --project src/Indexed.Cli -- find "SearchRequest" --glob "src/**/*.cs"

# Search extracted prose only
dotnet run --project src/Indexed.Cli -- find "lifetime" --mode prose --kind xml-doc

# Merge code + prose
dotnet run --project src/Indexed.Cli -- find "DisposeAsync" --mode auto

# Run against a non-git directory tree
dotnet run --project src/Indexed.Cli -- find "TargetId" --root C:\src\scratch

# Run against an explicit multi-root workspace
dotnet run --project src/Indexed.Cli -- find "OpenOrCreate" --root core=C:\src\proj\src --root docs=C:\src\proj\docs

# Check daemon status
dotnet run --project src/Indexed.Cli -- status

# List discovered daemon descriptors
dotnet run --project src/Indexed.Cli -- daemons

# Force a reconciliation rescan
dotnet run --project src/Indexed.Cli -- rescan

# Shut down the daemon
dotnet run --project src/Indexed.Cli -- stop

Requires the .NET 10 SDK. All projects target net10.0-windows.

Installing idx

From inside this repository, dotnet run is the simplest path (see Quick start).

To use Indexed in arbitrary repositories or directory workspaces, publish both the CLI and the daemon into the same directory and add it to PATH:

$dest = "$env:LOCALAPPDATA\\Programs\\Indexed"
dotnet publish src/Indexed.Cli -c Release -o $dest
dotnet publish src/Indexed.Service -c Release -o $dest

The CLI must be able to locate Indexed.Service.exe to start the daemon. If you cannot publish side-by-side, set INDEXED_SERVICE_EXE to the full path of Indexed.Service.exe (see docs/Indexed-Usage-Guide.md §1.3).

Documentation

• Tutorial — learning-oriented walkthrough for humans; read this first.
• Usage guide — CLI reference, HTTP API, configuration, data directory layout, troubleshooting.
• FAQ — short answers for common operational questions about target identity, status output, and freshness.
• Architecture — current-state architecture, layer ownership, data flow, concurrency model, failure handling.
• Workspace targets proposal — design record for the target model that now backs git, directory-tree, and directory-set indexing.
• Proposed improvements — prioritized next-step recommendations after workspace-target support landed.
• Index size reduction strategies — why trigram FTS5 is large and what can be done about it.
• Size reduction near-term plan — concrete “what to do next” plan for shrinking index.db safely.
• Architecture proposal — original design document (historical).
• Implementation plan — per-stage task breakdown and exit criteria (historical).

Workspace boundary

Indexed is an independent standalone repository with its own source, tests, and documentation lifecycle. It takes no dependency on any Near.* project; see the architecture proposal §6.2 for rationale.

Development guidance

• Run restore, build, test, and publish commands from the repository root with the real .NET SDK. global.json selects the .NET 10 SDK feature band used by the net10.0-windows projects.
• Production projects belong under src/, test projects under tests/, and current product documentation under docs/.
• Keep layer ownership explicit: target identity in Indexed.Targets, git integration in Indexed.Git, indexing and query execution in Indexed.Core, daemon hosting in Indexed.Service, and command-line behavior in Indexed.Cli.
• Generated bin/ and obj/ trees are ignored build outputs and must not be committed.
• README.md is the canonical repository and agent guidance; AGENTS.md and CLAUDE.md are symbolic links to it.
• Commit validated, self-contained changes directly to main, push them to origin/main, and include the repository's required Co-Authored-By trailer.

License

Repository contents are licensed under the MIT No Attribution License (MIT-0). See LICENSE.