feat: chatbot document rag logic advancement - #100
Merged
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthrough문서 정제·분류·청킹 파이프라인과 색인 스키마가 확장되었습니다. BM25·임베딩 대상 텍스트 필드 분리, 조회용 메타데이터/플래그 추가, 회귀 테스트 케이스 및 깃 설정(.gitignore·PR 템플릿) 변경이 포함됩니다. (총 변경: 스크립트·유틸·쿼리·테스트·설정 파일 다수 변경) Changes
Sequence Diagram(s)sequenceDiagram
participant Raw as Raw Document
participant Cleaner as Content Cleaner
participant Classifier as Doc Classifier
participant Chunker as Document Chunker
participant Indexer as Index Builder
participant BM25 as BM25 Indexer
participant Embedder as Embedding Generator
Raw->>Cleaner: 원문 전송
Cleaner->>Cleaner: 대시/ID 정규화<br/>편집자 푸터·다운로드 노이즈 제거
Cleaner->>Classifier: 정제된 텍스트 전달
Classifier->>Classifier: 제목/URL/내용 분석으로 문서 타입 분류
Classifier->>Chunker: 타입별 분할 로직 지시
Chunker->>Chunker: 세그먼트 생성 및 토큰 제한으로 청킹(오버랩 포함)
Chunker->>Indexer: 청크(부모ID, text_for_*, 메타데이터) 제출
Indexer->>BM25: `text_for_bm25` + 메타데이터로 색인
BM25->>BM25: 토큰화 및 역색인 구축
Indexer->>Embedder: `text_for_embedding`로 임베딩 생성 요청
Embedder->>Embedder: 임베딩 생성
Indexer->>Indexer: 최종 search_df와 doc_type_counts 완성
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 60 minutes.Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@LLM/sub_model/query_index.py`:
- Around line 328-335: The has_phone/has_email computation incorrectly treats
non-empty sentinel strings like "없음" as valid; update the logic in
query_index.py that builds has_phone and has_email from search_df (columns
"phone" and "email") to normalize and validate values: trim and lower the
string, reject known placeholders (e.g., "없음", "없다", "none", "n/a", "-", "na")
and only mark as present if the normalized value is not in that blacklist and
also matches a simple validation (phone: digits/optional separators of expected
length; email: basic local@domain pattern). Replace the current
.astype(str).ne("") checks with this combined blacklist + regex validation so
has_phone and has_email reflect true contact info.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 75e7c30d-0312-4497-92f9-6b3117236570
📒 Files selected for processing (6)
.github/pull_request_template.md.gitignoreLLM/sub_model/build_index.pyLLM/sub_model/index_utils.pyLLM/sub_model/query_index.pydebug/regression/chatbot_regression_cases.json
This was referenced May 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
관련 이슈
Close #99
🎯 배경
dmu_documents_cleaned.json의 실제 데이터 구조를 충분히 반영하지 못하고, 단순 문장/토큰 기준 청킹에 의존하고 있었습니다.🔍 주요 내용
dmu_documents_cleaned.json기반 RAG 전처리/청킹 구조 고도화fnctId/fnctNo, 다운로드 UI 텍스트 등 노이즈 제거intro,department,policy,history,privacy,contact,table_like,page문서 타입 분류 추가검색 인덱스 스키마 확장
text_for_embedding,text_for_bm25,text_for_answer분리breadcrumb,leaf_title,section_title,has_phone,has_date,has_credit,has_policy_keyword,is_privacy_old메타데이터 추가query_index.py검색/재랭킹 개선text컬럼으로 fallback회귀 테스트 케이스 추가
검증
build_index.py전체 실행 및 RAG 아티팩트 재생성search_df,embeddings,bm25row 수 정합성 확인변경 요약(1~3줄)
학교 웹사이트 RAG 파이프라인을 개선해 전처리·청킹·인덱스 스키마를 확장하고, 쿼리 의도 기반 검색 가중치와 재순위 로직을 도입해 검색·응답 정확도를 높였습니다.
주요 변경점
주의/리스크
다음 액션