SecondBrain Document Ingestion Pipeline

---

📋 Table of Contents

1. Project Overview
2. Phase 1: Document Ingestion
3. System Architecture
4. Installation & Setup
5. Usage Guide
6. Database Schema
7. Testing & Verification
8. Phase 2: Retrieval System
9. Phase 3: Visualization UI - TODO
10. Troubleshooting

---

Project Overview

SecondBrain is a document knowledge graph system that transforms PDFs into a queryable, semantic knowledge base. The system uses LLMs to extract structure, concepts, and insights, storing everything in a graph database with vector embeddings for semantic search.

Key Capabilities

• ✅ PDF document ingestion (any type: academic, technical, legal, etc.)
• ✅ Automatic structure extraction (sections, subsections)
• ✅ Concept extraction with importance levels
• ✅ Semantic embeddings for similarity search
• ✅ Entity recognition (authors, organizations)
• ✅ Graph-based relationship mapping
• ⏳ Q&A system (Phase 2)
• ⏳ Interactive visualization (Phase 3)

---

Phase 1: Document Ingestion

1. Enhanced Knowledge Graph Schema (KG/database.py)

New Collections:

• embeddings - 1024-dimensional vectors from Voyage AI
• extracted_concepts - Document-specific concepts with importance levels
• chunks - Semantic chunks for large sections

New Edge Types:

• has_embedding - Links sections/concepts to their embeddings
• summarizes - Links summaries to sections
• extracts_concept - Links sections to extracted concepts
• similar_to - Links similar concepts
• part_of - Concept hierarchies
• chunked_from - Links chunks to parent sections

2. Ingestion Pipeline Modules (ingestion/)

Module Structure:

ingestion/
├── __init__.py
├── pdf_processor.py        # PDF text extraction (PyMuPDF)
├── models.py               # Pydantic validation models
├── llm_analyzer.py         # Mistral AI document analysis
├── kg_populator.py         # Graph population logic
├── embedding_generator.py  # Voyage AI embeddings
└── pipeline.py             # End-to-end orchestration

Key Features:

• Retry logic with exponential backoff
• Model fallback (tries multiple Mistral models)
• Batch embedding generation
• Concept-based chunking with size fallback
• Comprehensive error handling

3. Dependencies Installed

dependencies = [
    "pyarango>=2.1.1",      # ArangoDB client
    "pymupdf>=1.24.0",      # PDF processing
    "mistralai>=1.0.0",     # LLM analysis
    "voyageai>=0.2.0",      # Embeddings
    "numpy>=1.26.0",        # Vector operations
    "pydantic>=2.5.0",      # Data validation
    "python-dotenv>=1.0.0", # Config management
]

Test Results

Document Processed: PICABench: How Far Are We from Physically Realistic Image Editing?

• 29 pages extracted
• 5 sections identified (Abstract, Introduction, Method, Experiment, Conclusion)
• 7 concepts extracted with importance levels
• 12 embeddings generated (5 section summaries + 7 concepts)
• 13 authors recognized
• Document type: Technical
• Domain: Computer Vision

Performance:

• PDF Processing: < 1 second
• LLM Analysis: ~1-2 minutes
• Graph Population: < 1 second
• Embedding Generation: ~5-10 seconds
• Total: ~2-3 minutes for 29-page document

---

System Architecture

Pipeline Flow

┌─────────────┐
│   PDF File  │
└──────┬──────┘
       │
       ▼
┌─────────────────────────────────────┐
│  1. PDF Processor (PyMuPDF)         │
│     - Extract text per page         │
│     - Extract metadata              │
└──────┬──────────────────────────────┘
       │
       ▼
┌─────────────────────────────────────┐
│  2. LLM Analyzer (Mistral AI)       │
│     - Identify sections             │
│     - Extract concepts              │
│     - Recognize entities            │
│     - Generate summaries            │
└──────┬──────────────────────────────┘
       │
       ▼
┌─────────────────────────────────────┐
│  3. KG Populator (ArangoDB)         │
│     - Create document node          │
│     - Create section nodes          │
│     - Create concept nodes          │
│     - Link all relationships        │
└──────┬──────────────────────────────┘
       │
       ▼
┌─────────────────────────────────────┐
│  4. Embedding Generator (Voyage AI) │
│     - Generate section embeddings   │
│     - Generate concept embeddings   │
│     - Link to source nodes          │
└──────┬──────────────────────────────┘
       │
       ▼
┌─────────────────────────────────────┐
│   Knowledge Graph (ArangoDB)        │
│   Ready for semantic search & Q&A   │
└─────────────────────────────────────┘

Data Flow

Document
    ├── Sections
    │   ├── Summary (with embedding)
    │   ├── Concepts (with embeddings)
    │   └── Chunks (if large)
    ├── Authors (Persons)
    └── Organizations

---

Installation & Setup

Prerequisites

1. Python 3.12+
2. ArangoDB (running on localhost:8529)
3. API Keys:
   • Mistral API key
   • Voyage AI API key

ArangoDB Setup

# Option 1: Homebrew (Mac)
brew install arangodb
brew services start arangodb

# Option 2: Docker
docker run -e ARANGO_NO_AUTH=1 -p 8529:8529 -d arangodb/arangodb:latest

Access at: http://localhost:8529

• Username: root
• Password: (empty)

Project Setup

# Clone/navigate to project
cd CONTRAnalysis

# Install dependencies
uv sync

# Create .env file with API keys
cat > .env << EOF
MISTRAL_API_KEY=your_mistral_key_here
VOYAGE_API_KEY=your_voyage_key_here
EOF

---

Usage Guide

Running the Pipeline

# Process a PDF document
uv run python main.py

By default, this processes technicalpaper.pdf. To process a different PDF:

uv run python main.py --pdf papername.pdf

Viewing Results

Option 1: ASCII Visualization

uv run python verify_database.py

This shows a complete tree structure of your documents with sections, concepts, and embeddings.

Option 2: ArangoDB Web UI

1. Go to http://localhost:8529
2. Login (username: root, password: empty)
3. IMPORTANT: Switch to documents_kg database (dropdown at top)
4. Click "COLLECTIONS" to browse
5. Click "QUERIES" to run AQL queries

Example Query:

FOR doc IN documents
  LET sections = (
    FOR section IN 1..1 OUTBOUND doc._id contains
    RETURN section.title
  )
  RETURN {
    title: doc.title,
    domain: doc.domain,
    sections: sections
  }

Programmatic Access

from KG.database import initialize_knowledge_graph

# Connect to database
db, collections, graph = initialize_knowledge_graph()

# Query documents
query = "FOR doc IN documents RETURN doc"
documents = list(db.AQLQuery(query, rawResults=True))

for doc in documents:
    print(f"{doc['title']} - {doc['domain']}")

# Find concepts
concepts_query = """
FOR concept IN extracted_concepts
  FILTER concept.importance == 'critical'
  RETURN {
    name: concept.name,
    description: concept.description
  }
"""
critical_concepts = list(db.AQLQuery(concepts_query, rawResults=True))

---

Database Schema

Collections

Collection	Purpose	Example Fields
documents	Main documents	title, content, document_type, domain, summary
sections	Document sections	title, content, section_type, position
subsections	Section subdivisions	title, summary, position
summaries	Section summaries	content, summary_type
extracted_concepts	Key concepts	name, description, importance, context
embeddings	Vector embeddings	vector (1024-dim), source_id, model
persons	Authors/people	name, identification, contact_info
organizations	Institutions	name, identification

Edge Collections

Edge	From → To	Purpose
contains	document → section	Hierarchical structure
summarizes	summary → section	Link summaries to content
extracts_concept	section → concept	Section-concept relationships
has_embedding	section/concept → embedding	Enable semantic search
authored_by	document → person	Author attribution

Graph Structure Example

Document: PICABench
    ├─[contains]→ Section: Abstract
    │               ├─[summarizes]← Summary: "PICABench is a benchmark..."
    │               ├─[extracts_concept]→ Concept: Physical Realism
    │               │                       └─[has_embedding]→ Embedding[1024]
    │               └─[has_embedding]→ Embedding[1024]
    │
    ├─[contains]→ Section: Introduction
    │               └─[extracts_concept]→ Concept: Instruction-based Editing
    │
    └─[authored_by]→ Person: Yihao Liu

---

Testing & Verification

Test Document Results

The system was successfully tested with technicalpaper.pdf (PICABench paper):

📖 PICABench: How Far Are We from Physically Realistic Image Editing?
├── 📄 Abstract (introduction)
│   ├── ✍️  Summary: "PICABench, a benchmark for evaluating..."
│   ├── 💡 PICAEval (high)
│   ├── 💡 Physical Realism (critical)
│   └── 🧮 1024-dim embedding
│
├── 📄 Introduction (introduction)
│   ├── 💡 Instruction-based Image Editing (high)
│   └── 🧮 1024-dim embedding
│
├── 📄 Method (methodology)
│   ├── 💡 Data Curation (medium)
│   ├── 💡 PICA-100K (high)
│   └── 🧮 1024-dim embedding
│
├── 📄 Experiment (results)
│   ├── 💡 Benchmark Results (high)
│   └── 🧮 1024-dim embedding
│
└── 📄 Conclusion (conclusion)
    ├── 💡 Future Work (medium)
    └── 🧮 1024-dim embedding

Statistics:

• 5 sections
• 7 concepts (varying importance: critical → medium)
• 12 embeddings (1024-dimensional vectors)
• 13 authors recognized
• All relationships properly linked

Verification Commands

# Run ASCII visualization
uv run python verify_database.py

# Check specific collections
uv run python -c "
from KG.database import initialize_knowledge_graph
db, _, _ = initialize_knowledge_graph()
print(f'Documents: {db[\"documents\"].count()}')
print(f'Sections: {db[\"sections\"].count()}')
print(f'Concepts: {db[\"extracted_concepts\"].count()}')
print(f'Embeddings: {db[\"embeddings\"].count()}')
"

---

Phase 2: Retrieval System

Overview

The retrieval system enables semantic search and question answering over the knowledge graph using RAG (Retrieval-Augmented Generation) pattern with advanced features.

Implemented Architecture

retrieval/
├── __init__.py              # Module exports
├── vector_search.py         # Vector similarity + hybrid search + caching
├── graph_retrieval.py       # Graph context gathering
├── qa_engine.py             # RAG pipeline + refinement + multi-doc QA
├── example_usage.py         # 8 comprehensive examples
└── README.md                # Full documentation

Key Features Implemented

1. Vector Search (retrieval/vector_search.py) ✅

Implemented Functions:

• search_by_embedding() - Core vector similarity search with caching
• search_similar_concepts() - Find similar concepts
• hybrid_search() - Combine vector (70%) + keyword (30%) ranking
• get_query_embedding() - Cached embedding generation
• cosine_similarity() - Efficient numpy-based similarity

Features:

• ✅ Automatic query embedding caching (in-memory, SHA-256 keyed)
• ✅ Configurable similarity thresholds
• ✅ Source type filtering (sections vs concepts)
• ✅ Document-specific search
• ✅ Hybrid ranking for better precision
• ✅ Batch processing for efficiency

2. Graph Retrieval (retrieval/graph_retrieval.py) ✅

Implemented Functions:

• get_context_for_section() - Comprehensive section context via graph traversal
• get_multi_hop_context() - Aggregate context from multiple sections
• get_concept_context() - Concept-specific context with relationships
• build_context_window() - Smart context formatting for LLM
• get_document_overview() - High-level document structure

Features:

• ✅ Configurable traversal depth (1-3 hops)
• ✅ Concept and entity extraction
• ✅ Neighbor section retrieval
• ✅ Insight inclusion
• ✅ Shared concept detection across sections
• ✅ Relationship tracking
• ✅ Token-aware context window building
• ✅ Summary prioritization for efficiency

3. Q&A Engine (retrieval/qa_engine.py) ✅

Implemented Functions:

• answer_question() - Main RAG pipeline with hybrid search
• refine_answer() - Multi-turn conversation support
• multi_document_qa() - Cross-document comparison and analysis
• build_rag_prompt() - Optimized prompt construction with citations

RAG Pipeline:

Question → Embed (Voyage AI) → Hybrid Search (Vector+Keyword) →
  → Graph Context Gathering → Prompt Construction →
  → LLM Generation (Mistral) → Answer + Sources + Confidence

Features:

• ✅ Hybrid search integration (vector + keyword)
• ✅ Graph context enrichment
• ✅ Citation tracking with section sources
• ✅ Confidence scoring (weighted by rank)
• ✅ Answer caching (in-memory, question-keyed)
• ✅ Multi-turn conversations with answer refinement
• ✅ Cross-document analysis and comparison
• ✅ Configurable temperature and top_k
• ✅ Error handling and graceful degradation

Example Usage

from retrieval.qa_engine import answer_question
import os

mistral_key = os.getenv("MISTRAL_API_KEY")
voyage_key = os.getenv("VOYAGE_API_KEY")

# Ask a question
result = answer_question(
    db=db,
    question="What is PICABench and how does it work?",
    mistral_key=mistral_key,
    voyage_key=voyage_key,
    top_k=3,
    use_hybrid_search=True
)

print(result["answer"])
print(f"\nConfidence: {result['confidence']:.3f}")
print(f"Cached: {result['cached']}")

print("\nSources:")
for source in result["sources"]:
    print(f"  {source['rank']}. {source['section_title']}")
    print(f"     Document: {source['document_title']}")
    print(f"     Similarity: {source['similarity']:.3f}")

Running Examples

# Run interactive examples
uv run python retrieval/example_usage.py

# Available examples:
# 1. Vector Similarity Search
# 2. Hybrid Search
# 3. Question Answering
# 4. Answer Refinement (multi-turn)
# 5. Graph Context Retrieval
# 6. Multi-Document Comparison
# 7. Concept Similarity Search
# 8. Document Overview

Performance Optimizations

Caching System:

• Query embeddings cached (SHA-256 keyed)
• Answers cached (question + doc_id keyed)
• Cache stats: get_cache_stats()
• Manual clear: clear_embedding_cache(), clear_answer_cache()

Smart Context Building:

• Automatic summary prioritization
• Token-aware window management (~4000 tokens max)
• Concept limiting (top 5 per section)
• Graceful truncation

Search Optimizations:

• Batch embedding generation
• Efficient numpy cosine similarity
• Configurable similarity thresholds
• Document-scoped filtering

---

Phase 3: Visualization UI

Goal

Create an interactive interface for exploring the knowledge graph and asking questions.

Planned Architecture

ui/
├── backend/
│   ├── __init__.py
│   ├── api.py           # FastAPI REST endpoints
│   └── websocket.py     # Real-time updates
└── frontend/
    ├── index.html
    ├── static/
    │   ├── css/
    │   ├── js/
    │   │   ├── graph.js     # D3.js visualization
    │   │   └── chat.js      # Chat interface
    │   └── lib/
    │       ├── d3.min.js
    │       └── vis.min.js
    └── components/
        ├── GraphView.js
        ├── ChatPanel.js
        └── DocumentTree.js

UI Layout Plan

┌─────────────────────────────────────────────────────────┐
│                     SecondBrain                          │
├──────────────────┬──────────────────────────────────────┤
│   📂 Documents   │                                      │
│   ├── PICABench  │         Graph Visualization          │
│   └── Climate ML │              (D3.js/vis.js)          │
│                  │                                      │
│   🔍 Search      │          [Interactive Graph]         │
│   [__________]   │            Nodes & Edges             │
│                  │          Click to explore            │
│   🏷️ Filters     │                                      │
│   □ Technical    │                                      │
│   □ Academic     ├──────────────────────────────────────┤
│   □ Legal        │   💬 Ask a Question                  │
│                  │   [_____________________________]    │
│   📊 Stats       │                                      │
│   5 docs         │   Q: What is PICABench?              │
│   23 concepts    │   A: PICABench is a benchmark for... │
│   45 embeddings  │                                      │
└──────────────────┴──────────────────────────────────────┘

Features to Implement

Backend (FastAPI)

# ui/backend/api.py
from fastapi import FastAPI
from retrieval.qa_engine import answer_question

app = FastAPI()

@app.get("/api/documents")
async def list_documents():
    """Get all documents"""
    pass

@app.get("/api/documents/{doc_id}/graph")
async def get_document_graph(doc_id: str):
    """Get graph data for visualization"""
    pass

@app.post("/api/query")
async def query_knowledge_graph(question: str):
    """Answer a question"""
    pass

@app.get("/api/concepts")
async def list_concepts(domain: str = None):
    """Get all concepts, optionally filtered"""
    pass

Frontend Features

1. Graph Visualization

• Interactive node-edge diagram
• Zoom and pan
• Click nodes to see details
• Highlight paths on query
• Color-code by type (document/section/concept)
• Size nodes by importance

2. Chat Interface

• Natural language questions
• Stream responses
• Show source citations
• Highlight relevant graph nodes
• Question history

3. Document Browser

• Tree view of all documents
• Filter by type/domain
• Search within documents
• View summaries
• Jump to sections

4. Concept Explorer

• List all extracted concepts
• Group by importance
• Show relationships
• Filter by domain
• Click to see where used

Technology Stack

Backend:

• FastAPI (REST API)
• WebSockets (real-time)
• Uvicorn (ASGI server)

Frontend:

• D3.js or vis.js (graph visualization)
• Vanilla JS or React (UI components)
• WebSocket client (real-time updates)

Integration:

Browser ←→ FastAPI ←→ Retrieval System ←→ ArangoDB
                 ↓
              Mistral AI (Q&A)
              Voyage AI (Search)

---

Troubleshooting

Common Issues

1. API Errors (502 Bad Gateway)

Problem: Mistral API returns 502 error
Solution: The code includes retry logic. Wait for automatic retry or check API status.

2. No Data in ArangoDB Web UI

Problem: Collections appear empty
Solution: Make sure you've switched to the documents_kg database (dropdown at top of page)

3. Import Errors

Problem: ModuleNotFoundError
Solution:

uv sync  # Reinstall dependencies

4. ArangoDB Connection Failed

Problem: Cannot connect to database
Solution:

# Check if ArangoDB is running
brew services list | grep arango
# Or
docker ps | grep arango

# Restart if needed
brew services restart arangodb

5. Embedding Generation Slow

Problem: Voyage AI requests timing out
Solution: This is normal for many embeddings. The code uses batch processing to optimize.

Debug Commands

# Verify database connection
uv run python -c "from KG.database import get_connection; conn = get_connection(); print('Connected!' if conn else 'Failed')"

# Check collections
uv run python verify_database.py

# Test PDF extraction only
uv run python -c "from ingestion.pdf_processor import extract_pdf_text; result = extract_pdf_text('technicalpaper.pdf'); print(f'Extracted {result[\"metadata\"][\"pages\"]} pages')"

---

File Structure Reference

CONTRAnalysis/
├── .env                          # API keys (not in git)
├── .clinerules/
│   └── conventions.md            # Coding conventions
├── KG/                           # Knowledge Graph module
│   ├── __init__.py
│   ├── database.py               # ✅ Schema & setup (ENHANCED)
│   ├── operations.py             # CRUD operations
│   ├── queries.py                # Complex graph queries
│   └── search.py                 # Search functionality
├── ingestion/                    # ✅ NEW - Ingestion pipeline
│   ├── __init__.py
│   ├── pdf_processor.py          # PDF extraction
│   ├── models.py                 # Pydantic models
│   ├── llm_analyzer.py           # Mistral analysis
│   ├── kg_populator.py           # Graph population
│   ├── embedding_generator.py    # Voyage embeddings
│   └── pipeline.py               # Orchestration
├── retrieval/                    # ✅ COMPLETE - Phase 2
│   ├── __init__.py
│   ├── vector_search.py          # Vector + hybrid search
│   ├── graph_retrieval.py        # Graph context gathering
│   ├── qa_engine.py              # RAG + refinement
│   ├── example_usage.py          # 8 examples
│   └── README.md                 # Full documentation
├── ui/                           # ⏳ TODO - Phase 3
│   ├── backend/
│   │   └── api.py
│   └── frontend/
│       └── index.html
├── main.py                       # ✅ Main entry point
├── verify_database.py            # ✅ Database verification
├── pyproject.toml                # ✅ Dependencies
├── PROGRESS.md                   # ✅ This file
└── technicalpaper.pdf            # Test document

---

Next Actions

Immediate (Phase 3)

1. Backend API

   • Set up FastAPI project
   • Implement REST endpoints
   • Add WebSocket support

2. Graph Visualization

   • Choose library (D3.js vs vis.js)
   • Implement basic graph rendering
   • Add interactivity

3. Chat Interface

   • Build chat UI
   • Connect to Q&A engine
   • Add source highlighting

---

Contact & Support

• Database: ArangoDB at http://localhost:8529 (database: documents_kg)
• Python Version: 3.12+
• Package Manager: uv

---

Last Updated: October 22, 2025
Phase 1 Status: ✅ COMPLETE
Phase 2 Status: ✅ COMPLETE
Next Phase: Phase 3 - Visualization UI