fix: rule_book LMM empty answer and BM25 search quality improvement - #86
Merged
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughLLM 규정집 검색·응답 로직을 조정합니다: 검색 청크 텍스트를 300자로 잘라 프롬프트 컨텍스트를 축소, 시스템 프롬프트를 엄격화하여 제공된 규정집만 사용하도록 제한, max_tokens 확대(512→1024), 응답의 finish_reason 및 빈 응답 로깅 추가, BM25용 청크 인덱스 텍스트 필드 도입으로 토크나이제이션 변경. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
LLM/rule_book/graph.py (1)
75-75:max_tokens는 환경변수로 분리하는 것을 권장합니다.Line 75 하드코딩 값은 운영 중 모델/트래픽 상황에 맞춘 즉시 튜닝이 어렵습니다.
제안 diff
TOP_K = int(os.getenv("RULE_BOOK_TOP_K", "5")) +RULE_BOOK_MAX_TOKENS = int(os.getenv("RULE_BOOK_MAX_TOKENS", "1024")) @@ - max_tokens=1024, + max_tokens=RULE_BOOK_MAX_TOKENS,🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@LLM/rule_book/graph.py` at line 75, The hardcoded max_tokens=1024 should be replaced with a configurable environment-backed value so it can be tuned without code changes; update the call that contains max_tokens=1024 to read an environment variable (e.g., os.getenv or a config loader), parse/validate it as an integer with a sensible default (e.g., 1024) and use that variable instead of the literal; ensure the change is applied where max_tokens is passed (the call containing max_tokens=1024) and add a brief validation/fallback to prevent non-integer or out-of-range values.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@LLM/rule_book/graph.py`:
- Around line 59-63: The code returns different "no results" messages depending
on the branch: one branch forces "해당 규정을 찾을 수 없습니다" (around the prompt
construction) while the early-return when chunks is empty (around the chunks
handling) uses a different phrase; unify them by introducing a single constant
(e.g., NO_RESULTS_MSG = "해당 규정을 찾을 수 없습니다") and use that constant both where you
currently early-return when chunks is empty and where you insert that message
into the prompt/template (references: the variable chunks and the
prompt-building logic in LLM/rule_book/graph.py).
In `@LLM/rule_book/index.py`:
- Around line 40-44: The current chunks.append call mixes BM25-weighted metadata
into the LLM prompt by duplicating source/article into the "text" field; instead
keep search-weighted text separate and make "text" contain only the actual
content for LLM. Change the dict in chunks.append to add a new key like
"bm25_text" or "search_text" that concatenates source and article_id for
retrieval weighting, keep "source" and "article" as metadata, and set "text" to
just content (or minimal context + content). Also update downstream usage (e.g.,
graph.py where the 300-char slice is applied) to use "text" for LLM context and
"bm25_text"/"search_text" for BM25/search operations.
---
Nitpick comments:
In `@LLM/rule_book/graph.py`:
- Line 75: The hardcoded max_tokens=1024 should be replaced with a configurable
environment-backed value so it can be tuned without code changes; update the
call that contains max_tokens=1024 to read an environment variable (e.g.,
os.getenv or a config loader), parse/validate it as an integer with a sensible
default (e.g., 1024) and use that variable instead of the literal; ensure the
change is applied where max_tokens is passed (the call containing
max_tokens=1024) and add a brief validation/fallback to prevent non-integer or
out-of-range values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: cc294ffb-9c7d-448f-b9a1-de42c02afed4
📒 Files selected for processing (2)
LLM/rule_book/graph.pyLLM/rule_book/index.py
Comment on lines
+59
to
+63
| "반드시 아래 <규정집 내용> 안의 텍스트만을 근거로 질문에 답하세요.\n" | ||
| "<규정집 내용> 외의 정보는 절대 사용하지 마세요.\n" | ||
| "답변은 간결하고 명확하게, 관련 조문 번호와 출처를 함께 안내하세요.\n" | ||
| "규정집에 없는 내용은 '해당 규정을 찾을 수 없습니다'라고 답하세요.\n" | ||
| "임의로 정보를 만들지 마세요." | ||
| "<규정집 내용>에 질문과 관련된 내용이 없으면 '해당 규정을 찾을 수 없습니다'라고만 답하세요.\n" | ||
| "추측하거나 임의로 정보를 만들지 마세요." |
There was a problem hiding this comment.
미검색 응답 문구를 분기 전체에서 동일하게 맞춰 주세요.
Line 62에서 "해당 규정을 찾을 수 없습니다"를 강제했는데, chunks가 비어 바로 반환되는 경로(Line 49)는 다른 문구를 사용합니다. 클라이언트가 고정 문구를 기준으로 처리하면 분기별 동작이 달라질 수 있습니다.
제안 diff
+NO_RULE_MSG = "해당 규정을 찾을 수 없습니다"
+
async def generate(state: RuleState) -> RuleState:
@@
- if not chunks:
- return {**state, "answer": "관련 규정을 찾지 못했습니다. 더 구체적인 키워드로 질문해 주세요."}
+ if not chunks:
+ return {**state, "answer": NO_RULE_MSG}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@LLM/rule_book/graph.py` around lines 59 - 63, The code returns different "no
results" messages depending on the branch: one branch forces "해당 규정을 찾을 수 없습니다"
(around the prompt construction) while the early-return when chunks is empty
(around the chunks handling) uses a different phrase; unify them by introducing
a single constant (e.g., NO_RESULTS_MSG = "해당 규정을 찾을 수 없습니다") and use that
constant both where you currently early-return when chunks is empty and where
you insert that message into the prompt/template (references: the variable
chunks and the prompt-building logic in LLM/rule_book/graph.py).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
관련 이슈
Open #85
🎯 배경
🔍 주요 내용
변경 요약
규정집 관련 챗봇의 타임아웃 문제를 해결하고 BM25 기반 검색 품질을 개선했습니다. LLM 토큰 한도를 늘리고 프롬프트 컨텍스트와 검색 색인을 조정했습니다.
주요 변경점
주의/리스크
다음 액션