feat: add rag retrieval evaluation dataset - #110
Conversation
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 Walkthrough워크스루이 PR은 졸업학점 관련 질의 감지를 위한 키워드/정규표현식을 확대하고, 답변 생성 시 재순위 신호를 다중화한 후, RAG 검색 및 답변 동작을 검증하는 회귀 테스트 인프라를 신규 도입합니다. 변경사항질의 분류 및 재순위 강화
RAG 회귀 테스트 인프라
추정 코드 검토 노력🎯 3 (보통) | ⏱️ ~25분 관련 PR
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
LLM/sub_model/query_index.py (1)
200-210: 💤 Low value
ql변수가 한국어 키워드에 대해 사실상 무의미한 중복 비교를 수행합니다
GRAD_KWS의 키워드는 모두 한국어 문자로 구성되어 있어.lower()적용 결과가 원문과 동일합니다. 따라서ql = text.lower()및any(k in text or k in ql ...)내부의or k in ql조건은 항상k in text와 동일한 결과를 반환합니다.♻️ 정리 제안
def _looks_like_grad_query(q: str) -> bool: text = q or "" if GRAD_NON_CREDIT_POLICY_RE.search(text) and not re.search(r"(학점|이수)", text): return False - ql = text.lower() compact = _compact(text) return ( bool(GRAD_CREDIT_QUERY_RE.search(text)) or ("졸업" in compact and "학점" in compact) - or any(k in text or k in ql for k in GRAD_KWS) + or any(k in text for k in GRAD_KWS) )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@LLM/sub_model/query_index.py` around lines 200 - 210, In _looks_like_grad_query the lowercase variable ql and the "or k in ql" check are redundant because GRAD_KWS contains only Korean tokens; remove ql = text.lower() and simplify the final any(...) clause to any(k in text for k in GRAD_KWS), leaving the rest of the logic unchanged (keep GRAD_NON_CREDIT_POLICY_RE, GRAD_CREDIT_QUERY_RE and compact checks intact) to eliminate the unnecessary duplicate membership test.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/regression/evaluate_rag_retrieval.py`:
- Line 228: The default output path is hardcoded to "/tmp/rag_eval_report.json"
which breaks portability and can expose files; change the
ap.add_argument("--out", ...) default to a cross-platform temp location by
importing tempfile and pathlib and using something like
Path(tempfile.gettempdir()) / "rag_eval_report.json" (or accept None and create
a secure temporary file with tempfile.NamedTemporaryFile when needed). Update
the ap.add_argument call (the "--out" argument) to use that computed path
instead of the literal "/tmp/..." and ensure imports for tempfile and
pathlib.Path are added.
In `@tests/regression/rag_eval_cases.json`:
- Around line 203-220: Two test entries have inconsistent URL IDs: for
"department_business_intro" expected_url_contains lists "4670/subview.do" but
answer_must_contain_any contains "4658/subview.do", and for
"department_visual_design_intro" expected_url_contains lists "4647/subview.do"
but answer_must_contain_any contains "4649/subview.do"; fix by making the URL
IDs consistent — either add the alternate IDs (4658 and/or 4649) into the
respective expected_url_contains arrays if they are valid alternative source
URLs, or remove the 4658/4649 entries from answer_must_contain_any if they were
erroneous, ensuring the JSON keys expected_url_contains and
answer_must_contain_any for the two IDs ("department_business_intro" and
"department_visual_design_intro") match.
---
Nitpick comments:
In `@LLM/sub_model/query_index.py`:
- Around line 200-210: In _looks_like_grad_query the lowercase variable ql and
the "or k in ql" check are redundant because GRAD_KWS contains only Korean
tokens; remove ql = text.lower() and simplify the final any(...) clause to any(k
in text for k in GRAD_KWS), leaving the rest of the logic unchanged (keep
GRAD_NON_CREDIT_POLICY_RE, GRAD_CREDIT_QUERY_RE and compact checks intact) to
eliminate the unnecessary duplicate membership test.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: efd3c49a-163c-49f6-b404-9afa5a5b67cc
📒 Files selected for processing (3)
LLM/sub_model/query_index.pytests/regression/evaluate_rag_retrieval.pytests/regression/rag_eval_cases.json
| ap.add_argument("--top-k", type=int, default=5, help="hybrid_search top_k for retrieval metrics") | ||
| ap.add_argument("--answer-top-k", type=int, default=6, help="build_answer top_k for answer metrics") | ||
| ap.add_argument("--schedule-top-k", type=int, default=5, help="schedule_search top_k for schedule cases") | ||
| ap.add_argument("--out", default="/tmp/rag_eval_report.json", help="output report path") |
There was a problem hiding this comment.
/tmp 하드코딩 경로는 플랫폼 호환성 및 보안 우려가 있습니다
Ruff S108이 지적하는 것처럼, /tmp는 Windows 환경에서는 기본적으로 존재하지 않아 portability 문제가 발생합니다. 또한 공유 CI 환경에서 /tmp/rag_eval_report.json 파일은 다른 사용자에게 노출될 수 있습니다.
🛡️ 수정 제안
+import tempfile
...
ap.add_argument(
"--out",
- default="/tmp/rag_eval_report.json",
+ default=str(Path(tempfile.gettempdir()) / "rag_eval_report.json"),
help="output report path",
)🧰 Tools
🪛 Ruff (0.15.12)
[error] 228-228: Probable insecure usage of temporary file or directory: "/tmp/rag_eval_report.json"
(S108)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/regression/evaluate_rag_retrieval.py` at line 228, The default output
path is hardcoded to "/tmp/rag_eval_report.json" which breaks portability and
can expose files; change the ap.add_argument("--out", ...) default to a
cross-platform temp location by importing tempfile and pathlib and using
something like Path(tempfile.gettempdir()) / "rag_eval_report.json" (or accept
None and create a secure temporary file with tempfile.NamedTemporaryFile when
needed). Update the ap.add_argument call (the "--out" argument) to use that
computed path instead of the literal "/tmp/..." and ensure imports for tempfile
and pathlib.Path are added.
| "id": "department_business_intro", | ||
| "category": "department", | ||
| "kind": "search", | ||
| "query": "경영학과 소개", | ||
| "expected_url_contains": ["4670/subview.do"], | ||
| "expected_title_contains": ["경영학과"], | ||
| "answer_must_contain_any": ["경영학과", "4658/subview.do"], | ||
| "requires_source_url": true | ||
| }, | ||
| { | ||
| "id": "department_visual_design_intro", | ||
| "category": "department", | ||
| "kind": "search", | ||
| "query": "시각디자인과 소개", | ||
| "expected_url_contains": ["4647/subview.do"], | ||
| "expected_title_contains": ["시각디자인과"], | ||
| "answer_must_contain_any": ["시각디자인과", "4649/subview.do"], | ||
| "requires_source_url": true |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's check the test file structure and see the actual content around lines 203-220
cat -n tests/regression/rag_eval_cases.json | sed -n '200,225p'Repository: dongsooop/AI
Length of output: 979
🏁 Script executed:
# Also, let's look at the full context of all department test cases
grep -n "department_" tests/regression/rag_eval_cases.json | head -20Repository: dongsooop/AI
Length of output: 295
🏁 Script executed:
# Now let's find and examine the evaluation logic that handles these fields
find . -type f -name "*.py" -o -name "*.js" -o -name "*.ts" | head -20Repository: dongsooop/AI
Length of output: 645
🏁 Script executed:
# Search for the evaluate_search_case function mentioned in the review
rg "evaluate_search_case|source_url_match|answer_keyword_match" --type py -A 5 -B 2Repository: dongsooop/AI
Length of output: 7025
🏁 Script executed:
# Get more context around the evaluate_search_case function
sed -n '1,100p' tests/regression/evaluate_rag_retrieval.pyRepository: dongsooop/AI
Length of output: 3489
🏁 Script executed:
# Look at the full evaluate_search_case function
sed -n '20,80p' tests/regression/evaluate_rag_retrieval.pyRepository: dongsooop/AI
Length of output: 2307
🏁 Script executed:
# Get line count to see the file size
wc -l tests/regression/evaluate_rag_retrieval.pyRepository: dongsooop/AI
Length of output: 103
🏁 Script executed:
# Get the rest of the evaluate_search_case function
sed -n '60,120p' tests/regression/evaluate_rag_retrieval.pyRepository: dongsooop/AI
Length of output: 2751
🏁 Script executed:
# Let's also check the other department test cases to understand the pattern
sed -n '183,230p' tests/regression/rag_eval_cases.jsonRepository: dongsooop/AI
Length of output: 1674
🏁 Script executed:
# Verify the logic one more time by checking the full evaluate_search_case function
sed -n '85,105p' tests/regression/evaluate_rag_retrieval.pyRepository: dongsooop/AI
Length of output: 1179
expected_url_contains와 answer_must_contain_any 간 URL ID 불일치 – 데이터 정합성 문제
두 케이스에서 URL ID가 불일치합니다:
department_business_intro: expected_url_contains4670/subview.dovs answer_must_contain_any4658/subview.dodepartment_visual_design_intro: expected_url_contains4647/subview.dovs answer_must_contain_any4649/subview.do
평가 로직에서 source_url_match는 expected_url_contains를 기준으로 검증되지만, answer_must_contain_any의 URL(4658, 4649)은 실제로 검증되지 않습니다. contains_any()의 OR 로직상 한글 키워드("경영학과", "시각디자인과")가 먼저 일치하기 때문입니다. 이는 해당 URL들이 검증 대상인 것처럼 보이나, 실제로는 검증되지 않는 "죽은" 데이터입니다.
일관성을 위해 다음 중 하나를 고려하세요:
- 4658/4649가 유효한 대체 URL이면
expected_url_contains에 추가 - 단순 오류면
answer_must_contain_any에서 제거
다른 department 케이스들(department_computer_engineering_intro, department_software_intro)은 URL이 일관되게 설정되어 있습니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/regression/rag_eval_cases.json` around lines 203 - 220, Two test
entries have inconsistent URL IDs: for "department_business_intro"
expected_url_contains lists "4670/subview.do" but answer_must_contain_any
contains "4658/subview.do", and for "department_visual_design_intro"
expected_url_contains lists "4647/subview.do" but answer_must_contain_any
contains "4649/subview.do"; fix by making the URL IDs consistent — either add
the alternate IDs (4658 and/or 4649) into the respective expected_url_contains
arrays if they are valid alternative source URLs, or remove the 4658/4649
entries from answer_must_contain_any if they were erroneous, ensuring the JSON
keys expected_url_contains and answer_must_contain_any for the two IDs
("department_business_intro" and "department_visual_design_intro") match.
1 participant
관련 이슈
Open #109
🎯 배경
복학,현장실습,졸업보류,졸업유예,전문학사 졸업학점처럼 검색 후보는 잡히지만 최종 답변 출처가 어긋나는 케이스가 확인되어, 검색 결과 재정렬 규칙도 함께 보정했습니다.🔍 주요 내용
RAG 평가 자동화 스크립트 추가
tests/regression/evaluate_rag_retrieval.pyRAG 평가셋 30개 추가
tests/regression/rag_eval_cases.json검색/답변 재정렬 규칙 개선
졸업보류,졸업유예,졸업연기는 학점 질문이 아니면 졸업학점 추출 로직을 타지 않도록 분리title,leaf_title,breadcrumb의 핵심 토큰 매칭 점수를 최종 답변 후보 정렬에 반영현장실습 안내처럼 부서/센터가 아닌 학사안내 페이지가 우선되어야 하는 정책성 질문 보정전문학사 졸업학점질의에서 전문학사 페이지가 우선되도록 졸업학점 검색 재정렬 보정검증 결과
변경 요약
RAG 검색 성능을 평가하기 위한 자동화된 평가 스크립트와 30개의 테스트 케이스를 추가했습니다. 동시에 졸업학점, 정책 검색 등에 대한 답변 재순위 로직을 개선하여 더 정확한 검색 결과를 제공합니다.
주요 변경점
evaluate_rag_retrieval.py): Top-1/Top-3 URL 정확도, 제목 매칭, 할루시네이션 감지 등 8가지 메트릭으로 RAG 성능 측정rag_eval_cases.json): 연락처, 정책, 졸업, 학과, 기숙사, 개인정보, 학사일정 등 7개 카테고리의 30개 테스트 케이스주의/리스크
다음 액션