Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added .DS_Store
Binary file not shown.
18 changes: 14 additions & 4 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,33 +1,43 @@
# CodeRag Environment Configuration
# Copy this file to .env and fill in your actual values
# The .env file should NEVER be committed to git (see .gitignore)
# CodeRag loads .env from the current working directory automatically.

# ============================================
# GEMINI EMBEDDING API KEY
# ============================================
# Required when using Gemini embedding provider (CODERAG_EMBEDDING_PROVIDER=gemini)
# Get your API key from: https://makersuite.google.com/app/apikey
CODERAG_GEMINI_API_KEY=your_api_key_here
# Compatibility alias also accepted: CODERAG_GEMINI_AI_KEY

# Optional: Override the default Gemini embedding model
# Default: models/gemini-embedding-2-preview
CODERAG_GEMINI_MODEL=models/gemini-embedding-2-preview
# Default: models/gemini-embedding-001
CODERAG_GEMINI_MODEL=models/gemini-embedding-001

# ============================================
# EMBEDDING CONFIGURATION
# ============================================
# Choose embedding provider: "local-hash" (free, offline) or "gemini" (better quality, requires API key)
# Choose embedding provider: "local-hash" (free, offline), "gemini" (better quality, requires API key), or "onnx" (local neural embeddings via @xenova/transformers)
# Default: local-hash
CODERAG_EMBEDDING_PROVIDER=gemini

# Dimensions for local-hash embeddings (ignored for Gemini which always uses 768)
# Dimensions for local-hash embeddings (ignored for Gemini which explicitly requests 768, or ONNX which uses 384)
# Default: 256
# CODERAG_EMBEDDING_DIMENSIONS=256

# Timeout for embedding API calls in milliseconds
# Default: 30000
# CODERAG_EMBEDDING_TIMEOUT_MS=30000

# ============================================
# ONNX EMBEDDING CONFIGURATION (provider=onnx)
# ============================================
# Directory containing the Xenova/gte-small model (relative to CWD or absolute path)
# The model should be at <onnxModelDir>/Xenova/gte-small/ with tokenizer.json, config.json, and onnx/ subdirectory
# Default: .coderag-models/models
# CODERAG_ONNX_MODEL_DIR=.coderag-models/models

# ============================================
# DIRECT LLM CONFIGURATION (Optional)
# ============================================
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,6 @@ node_modules
.coderag*
coverage
.env
.qwen/
.serena/
*.tgz
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,6 @@ New indexing, retrieval, transport, or MCP behavior must include direct coverage

8. Document operator setup.
Any required setup for local model servers, storage locations, or git hooks must be reflected in `README.md`.

9. Preserve future-ready features behind flags.
If a feature is correctly implemented but blocked by external platform constraints (not code errors), gate it behind an optional config flag rather than removing it. This keeps the codebase ready for when platform support arrives. Document the flag and its current support status in `README.md`.
47 changes: 43 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# CodeRag

CodeRag is a standalone npm package that gives coding agents targeted retrieval over a JavaScript or TypeScript repository. It uses `@abhinav2203/codeflow-core` for repo analysis, stores node documents in LanceDB, traverses graph edges for surrounding context, and can optionally ask a local LLM server to turn the retrieved context into an answer.
CodeRag is a standalone npm package that gives coding agents targeted retrieval over a codebase. It uses `@abhinav2203/codeflow-core` with tree-sitter for multi-language repo analysis, stores node documents in LanceDB, traverses graph edges for surrounding context, and can optionally ask a local LLM server to turn the retrieved context into an answer.

**Supported languages:** TypeScript, JavaScript, Go, Python, C, C++, Rust.

## What ships in this repo

Expand Down Expand Up @@ -65,23 +67,56 @@ CodeRag loads configuration in this order:
1. Explicit `--config` path
2. `coderag.config.json`
3. `.coderag.json`
4. Environment overrides
4. `.env` values from the current working directory
5. Environment overrides

Supported environment overrides:

- `CODERAG_REPO_PATH`
- `CODERAG_STORAGE_ROOT`
- `CODERAG_EMBEDDING_PROVIDER`
- `CODERAG_EMBEDDING_DIMENSIONS`
- `CODERAG_ONNX_MODEL_DIR`
- `CODERAG_GEMINI_MODEL`
- `CODERAG_GEMINI_API_KEY`
- `CODERAG_GEMINI_AI_KEY`
- `CODERAG_EMBEDDING_TIMEOUT_MS`
- `CODERAG_TOP_K`
- `CODERAG_RERANK_K`
- `CODERAG_MAX_CONTEXT_CHARS`
- `CODERAG_DEFAULT_DEPTH`
- `CODERAG_MAX_DEPTH`
- `CODERAG_LOCK_TIMEOUT_MS`
- `CODERAG_LOCK_POLL_MS`
- `CODERAG_LOCK_STALE_MS`
- `CODERAG_SERVICE_HOST`
- `CODERAG_SERVICE_PORT`
- `CODERAG_SERVICE_API_KEY`
- `CODERAG_LLM_ENABLED`
- `CODERAG_LLM_TRANSPORT`
- `CODERAG_LLM_BASE_URL`
- `CODERAG_LLM_MODEL`
- `CODERAG_LLM_API_KEY`
- `CODERAG_LLM_TIMEOUT_MS`
- `CODERAG_CUSTOM_HTTP_FORMAT`
- `CODERAG_LLM_HEADERS`

When `embedding.provider` is `gemini`, CodeRag defaults to `models/gemini-embedding-001` and requests 768-dimensional vectors explicitly so the stored embedding fingerprint matches the vectors written to LanceDB. It accepts either `CODERAG_GEMINI_API_KEY` or the compatibility alias `CODERAG_GEMINI_AI_KEY`.

When `embedding.provider` is `onnx`, CodeRag uses `Xenova/gte-small` (384-dim, ~33MB) running locally via `@xenova/transformers`. No API key or external server needed. The model must be downloaded to `<onnxModelDir>/Xenova/gte-small/` (default `.coderag-models/models/Xenova/gte-small/`).

```bash
Comment on lines +104 to +108

Copilot AI Apr 8, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README says the ONNX embedding provider uses Xenova/gte-small, but the implementation defaults to Xenova/all-MiniLM-L6-v2 (and tests assert that). Please align the README model name + download snippet with the actual default (or change the provider default) so users download the correct model.

Copilot uses AI. Check for mistakes.
# Download the ONNX embedding model (~33MB)
python3 -c "
from huggingface_hub import snapshot_download
snapshot_download('Xenova/gte-small', local_dir='.coderag-models/models',
allow_patterns=['onnx/model_quantized.onnx', 'config.json',
'tokenizer.json', 'tokenizer_config.json',
'special_tokens_map.json'])
"

# Then set embedding.provider to "onnx" in your config and run coderag init
```

## Local LLM integration

Expand Down Expand Up @@ -171,6 +206,7 @@ coderag index [--config path]
coderag reindex [--config path] [--full]
coderag query "question" [--config path] [--depth 2] [--json]
coderag serve-mcp [--config path]
coderag serve-http [--config path]
coderag doctor [--config path]
```

Expand All @@ -180,9 +216,12 @@ coderag doctor [--config path]

## Production notes

- JavaScript and TypeScript repos are supported.
- TypeScript, JavaScript, Go, Python, C, C++, and Rust repos are supported.
- Excluded directories: `node_modules`, `.git`, `.next`, `dist`, `build`, `target`, `__pycache__`, `vendor`, `.venv`, `artifacts`, `coverage`.
- Call-site extraction is best effort for dynamic dispatch, reflection, or generated code. Missing call sites are returned as unresolved metadata, not guessed values.
- The built-in embedding strategy is deterministic and zero-setup. If you need stronger semantic recall, provide a custom embedding provider through the library API.
- The built-in `local-hash` embedding strategy is deterministic and zero-setup. The `onnx` provider runs `Xenova/gte-small` locally (384-dim, ~33MB) for semantic-quality embeddings without any API key. If you need cloud-quality embeddings, use the `gemini` provider.
- `serve-http` exposes `/health`, `/ready`, `/metrics`, and `/v1/*` endpoints. `/ready` only reports ready once the index exists, contains documents, and matches the configured embedding fingerprint.
- If you use Gemini embeddings, set `CODERAG_GEMINI_API_KEY` or `CODERAG_GEMINI_AI_KEY` before indexing. Changing `CODERAG_GEMINI_MODEL` requires a full reindex because the persisted embedding fingerprint includes the model name and dimensions.
- Live E2E runs in this repo were verified against an OpenAI-compatible NVIDIA endpoint and against both the CodeRag and CodeFlow repositories.

## Development
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# 🎯 Post-Commit Quality Gate Report

**Commit:** 971d68d feat: add Gemini and ONNX embedding providers
**Date:** 2026-04-06T15:24:54+05:30
**Author:** Abhinav Nehra <nehraa@student.wpunj.edu>
**Branch:** feat/gemini-onnx-embedding-providers

---

## 📊 Summary

| Metric | Value |
|--------|-------|
| Changed Files | 0 |
| Source Files | 0 |
| Test Files | 0 |
| Doc Files | 0 |

---

## 🎯 Quality Gate Results

| Stage | Status | Details |
|-------|--------|---------|

| /7 | Linting & Code Quality | PASS | Checked 1 files |
| /7 | Security Analysis | FAIL | Scanned for secrets, injections, dependencies |
| /7 | Fix Security Issues | PASS | Fixed 0 issues |
| /7 | Run Existing Tests | FAIL | Ran test suite |
| /7 | Add/Update Tests | PASS | Identified 0 files |
| /7 | Update Documentation | PASS | Checked README, CHANGELOG, inline docs |
| /7 | Context Compaction | PASS | Compacted from 40K to 40K |

---

## 📁 Detailed Reports

- [Stage 1: Linting](stage-01-linting.md)
- [Stage 2: Security](stage-02-security.md)
- [Stage 3: Fix Security](stage-03-fix-security.md)
- [Stage 4: Run Tests](stage-04-run-tests.md)
- [Stage 5: Add Tests](stage-05-add-tests.md)
- [Stage 6: Documentation](stage-06-documentation.md)
- [Stage 7: Context](stage-07-context.md)

---

## ✅ Next Steps

1. **Fix any FAIL statuses** above
2. **Review security issues** and apply fixes
3. **Add tests** for new functionality
4. **Update documentation** for changed APIs
5. **Commit fixes** to trigger another quality gate

---

*Generated by post-commit quality gate hook*
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Post-Commit Quality Gate Summary

**Commit:** 971d68d feat: add Gemini and ONNX embedding providers
**Date:** 2026-04-06T15:24:54+05:30
**Changed Files:** 0

## Quality Gate Results

| Stage | Status | Details |
|-------|--------|---------|

| /7 | Linting & Code Quality | PASS | Checked 1 files |
| /7 | Security Analysis | FAIL | Scanned for secrets, injections, dependencies |
| /7 | Fix Security Issues | PASS | Fixed 0 issues |
| /7 | Run Existing Tests | FAIL | Ran test suite |
| /7 | Add/Update Tests | PASS | Identified 0 files |
| /7 | Update Documentation | PASS | Checked README, CHANGELOG, inline docs |

## Key Takeaways
- Review any FAIL statuses above
- Fix security issues before next commit
- Add tests for new functionality
- Update documentation as needed
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Stage 1: Linting & Code Quality

**Status:** PASS
**Files Checked:** 1

✅ No linting issues found
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Stage 2: Security Analysis

**Status:** FAIL


### npm Audit Vulnerabilities
```
npm warn config production Use `--omit=dev` instead.
found 0 vulnerabilities
```

## Security Checks Performed
- ✅ Hardcoded secrets scan
- ✅ SQL injection risks
- ✅ eval/exec usage
- ✅ Dependency vulnerabilities
- ✅ XSS patterns
- ✅ Path traversal risks
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Stage 3: Fix Security Issues

**Status:** PASS
**Issues Fixed:** 0

✅ No security issues required fixing

## Auto-Fixes Applied
- Hardcoded secrets → Environment variables
- SQL injection → Parameterized queries (manual review needed)
- eval/exec → Safer alternatives (manual review needed)
Loading