Feature/rag/#37 by Yoo-SH · Pull Request #46 · OpenContextAI/open-context

Yoo-SH · 2025-12-15T08:04:33Z

📋 요약

RAG 시스템을 FastAPI로 분리하고, 로컬 개발 환경을 개선하며, 파일명 기반 검색 기능을 강화했습니다.

✨ 주요 변경사항

1. RAG 서비스 분리 및 리팩토링

FastAPI 기반 RAG 서비스 분리
LangChain 통합
Elasticsearch를 활용한 키워드 + 벡터 하이브리드 검색
검색 가중치: 키워드 30% / 벡터 70%

2. 로컬 개발 환경 구축

Elasticsearch HTTP 모드 설정 (SSL 비활성화)
임베딩 모델 통합 (bge-m3:latest, 1024차원)
개발용 설정 파일 정리

3. MCP 서버 통합

Claude Code와 MCP 서버 연동
find_knowledge 및 get_content 도구 제공
HTTP 모드 설정 (localhost:3000)

4. 문서화 개선

QUICKSTART.md: 5분 빠른 시작 가이드
CLAUDE.md: Claude Code를 위한 프로젝트 가이드
.env.example: 환경 변수 템플릿
README.md: QUICKSTART 링크 추가

5. 검색 기능 강화

파일명 기반 검색 추가 (metadata.originalFilename)
검색 필드 가중치 최적화:
- 파일명: 3.0배 (최우선)
- 본문: 2.0배
- 제목: 1.5배

6. 의존성 정리

Unstructured API 제거 (미사용)
docker-compose.yml 정리

🔧 기술 스택

Backend: Spring Boot 3.3.11 + Java 21
RAG Service: FastAPI + LangChain
Search: Elasticsearch (BM25 + Vector)
Embedding: Ollama bge-m3:latest (1024-dim)
MCP: Node.js MCP Adapter

📊 검색 개선 효과

Before:

검색 필드: content, title (2개)
파일명 매칭 시 낮은 순위

After:

검색 필드: originalFilename, content, title (3개)
파일명 정확 매칭 시 최우선 순위 (3배 가중치)

예시:

검색: "Spring Security Guide"
- Spring_Security_Guide.pdf → 1위 (파일명 매칭)
- API_Documentation.pdf → 2위 (내용 매칭)

🧪 테스트

로컬 환경에서 Elasticsearch 연결 확인
MCP 서버 연동 테스트
파일명 기반 검색 테스트
하이브리드 검색 정확도 확인
문서화 검증

📝 관련 이슈

[BACKEND] RAG서버를 SPIRNG에서 분리하기 #37
Closes [BACKEND] RAG서버를 SPIRNG에서 분리하기 #37

🚀 배포 후 작업

기존 문서 재인덱싱 (파일명 검색 적용)
프로덕션 환경 Elasticsearch 설정 검토
사용자 피드백 수집

리뷰 포인트:

RAG 서비스 분리 아키텍처
파일명 기반 검색 가중치 적절성
문서화 완성도
로컬 개발 환경 설정 방식

- rag pipeline fastapi에서 처리 - langchain과 결합 - elasticsearch 데이터 저장 for(keyword, vector)

- 백엔드에서는 rag 검색만 진행 - 키워드(파일원본,제목,본문)검색+백터검색 3/7 - unstructed io 관련설정 제거

Copilot

Pull request overview

This PR implements a FastAPI-based RAG (Retrieval-Augmented Generation) service separation, migrating document processing from the Java Spring Boot core service to a dedicated Python microservice. The implementation includes hybrid search (BM25 + vector similarity), filename-based search enhancement, and improved local development setup.

Key Changes:

Separates RAG pipeline into standalone FastAPI service with LangChain and Ollama integration
Adds filename-based search field with 3x weighting for improved document retrieval
Implements paragraph-based chunking and PyMuPDF document parsing
Configures local development environment with HTTP-mode Elasticsearch

Reviewed changes

Copilot reviewed 37 out of 61 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
rag/requirements.txt	Python dependencies for FastAPI RAG service
rag/app/services/*.py	Core RAG services: parsing, chunking, embedding, indexing, and search
rag/app/api/v1/*.py	FastAPI endpoints for document processing, search, and content retrieval
rag/app/config/settings.py	Pydantic-based configuration management
rag/Dockerfile	Container definition for RAG service
docker-compose.yml	Replaces unstructured-api with FastAPI RAG service
core/src/main/java/com/opencontext/service/SearchService.java	Adds filename field to search query with 3x boost
core/src/main/java/com/opencontext/service/IndexingService.java	Stores originalFilename in metadata for search
core/src/main/java/com/opencontext/controller/SourceController.java	Integrates with RAG service via REST API
core/src/main/resources/application*.yml	Configuration updates for RAG service integration
core/src/main/java/com/opencontext/config/SecurityConfig.java	Temporarily disables API authentication

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-15T08:14:06Z

+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],  # In production, restrict to specific origins
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)


The CORS middleware is configured to allow all origins with 'allow_origins=["*"]'. While the comment acknowledges this should be restricted in production, having permissive CORS in code increases the risk it will be deployed to production. Consider using environment-based configuration to enforce proper CORS settings.

Copilot · 2025-12-15T08:14:07Z

+
+    Pipeline:
+    1. Download file from presigned URL
+    2. Parse document (Unstructured.io)


The comment states 'Parse document (Unstructured.io)' but the code is actually using PyMuPDF for parsing. This is misleading and should be updated to reflect the actual implementation.

Suggested change

2. Parse document (Unstructured.io)

2. Parse document (PyMuPDF)

Copilot · 2025-12-15T08:14:07Z

+    title: str = Field(
+        ...,
+        description="Section heading or title (from H1 header)"


The field description states 'Section heading or title (from H1 header)' but the actual implementation extracts the title from the first line or first 50 characters of a paragraph, not from H1 headers. This misleading documentation should be updated to match the actual behavior.

Copilot · 2025-12-15T08:14:07Z

+    elementType: str = Field(
+        default="TitleBasedChunk",
+        description="Type of chunk (TitleBasedChunk for H1-split chunks)"
+    )


The field description states 'Type of chunk (TitleBasedChunk for H1-split chunks)' with a default value of 'TitleBasedChunk', but the actual implementation uses 'ParagraphChunk' as the element type (line 88 in chunking.py). This inconsistency between the model default and actual usage should be corrected.

Copilot · 2025-12-15T08:14:08Z

+  profiles:
+    active: dev
+
+  # Database Configuration (Development)


The comment incorrectly describes the configuration as 'Docker' when it's actually for local development based on the values (localhost URLs). The profile is also set to 'dev' on line 6. Update the comment to accurately reflect this is a development configuration.

Copilot · 2025-12-15T08:14:10Z

+import logging
+
+from app.config.settings import settings
+from app.utils.exceptions import RagServiceException, rag_exception_handler


Import of 'rag_exception_handler' is not used.

Copilot · 2025-12-15T08:14:10Z

+Internal API called by core/ service after file upload.
+"""
+import logging
+from fastapi import APIRouter, HTTPException


Import of 'HTTPException' is not used.

Suggested change

from fastapi import APIRouter, HTTPException

from fastapi import APIRouter

Copilot · 2025-12-15T08:14:11Z

+from app.models.responses import ProcessResponse
+from app.services.parsing import DocumentParsingService
+from app.services.chunking import ChunkingService
+from app.services.embedding import EmbeddingService


Import of 'EmbeddingService' is not used.

Suggested change

from app.services.embedding import EmbeddingService

Copilot · 2025-12-15T08:14:11Z

+from app.services.parsing import DocumentParsingService
+from app.services.chunking import ChunkingService
+from app.services.embedding import EmbeddingService
+from app.services.indexing import IndexingService


Import of 'IndexingService' is not used.

Suggested change

from app.services.indexing import IndexingService

Copilot · 2025-12-15T08:14:11Z

+Provides accurate token counting for content retrieval with truncation support.
+"""
+import tiktoken
+from typing import Optional


Import of 'Optional' is not used.

kAYI0019 · 2025-12-15T08:48:50Z

RAG 리팩토링, 검색 정확도 개선, 문서화까지 고생하셨습니다!

Yoo-SH added 3 commits December 15, 2025 16:50

refactro(#37): rag서비스 분리

9000ea2

- rag pipeline fastapi에서 처리 - langchain과 결합 - elasticsearch 데이터 저장 for(keyword, vector)

refactor(#37): 백엔드 rag 리팩토링

08279e2

- 백엔드에서는 rag 검색만 진행 - 키워드(파일원본,제목,본문)검색+백터검색 3/7 - unstructed io 관련설정 제거

build(#37): 도커 컴포즈 파일 업데이트

527c0a9

Yoo-SH requested review from Copilot, kAYI0019 and qowlgur121 December 15, 2025 08:04

Yoo-SH self-assigned this Dec 15, 2025

Yoo-SH added backend refactor rag labels Dec 15, 2025

Copilot started reviewing on behalf of Yoo-SH December 15, 2025 08:04 View session

Merge branch 'main' into feature/rag/#37

5688122

Copilot AI reviewed Dec 15, 2025

View reviewed changes

qowlgur121 mentioned this pull request Dec 15, 2025

[BUG] PR #46 설정 불일치로 Docker 환경 실행 불가 #48

Open

kAYI0019 approved these changes Dec 15, 2025

View reviewed changes

qowlgur121 mentioned this pull request Dec 15, 2025

fix(docker): Docker 배포 설정 오류 수정 #49

Merged

Yoo-SH merged commit 06eac29 into main Dec 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature/rag/#37#46

Feature/rag/#37#46
Yoo-SH merged 4 commits into
mainfrom
feature/rag/#37

Yoo-SH commented Dec 15, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 15, 2025

Uh oh!

Copilot AI Dec 15, 2025

Uh oh!

Copilot AI Dec 15, 2025

Uh oh!

Copilot AI Dec 15, 2025

Uh oh!

Copilot AI Dec 15, 2025

Uh oh!

Copilot AI Dec 15, 2025

Uh oh!

Copilot AI Dec 15, 2025

Uh oh!

Copilot AI Dec 15, 2025

Uh oh!

Copilot AI Dec 15, 2025

Uh oh!

Copilot AI Dec 15, 2025

Uh oh!

kAYI0019 commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	2. Parse document (Unstructured.io)
	2. Parse document (PyMuPDF)

	from fastapi import APIRouter, HTTPException
	from fastapi import APIRouter

Uh oh!

Conversation

Yoo-SH commented Dec 15, 2025

📋 요약

✨ 주요 변경사항

1. RAG 서비스 분리 및 리팩토링

2. 로컬 개발 환경 구축

3. MCP 서버 통합

4. 문서화 개선

5. 검색 기능 강화

6. 의존성 정리

🔧 기술 스택

📊 검색 개선 효과

🧪 테스트

📝 관련 이슈

🚀 배포 후 작업

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

kAYI0019 commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants