feat: add rag quality evaluation question set - #120
Merged
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughPR 변경 사항RAG 회귀 검증 인프라 및 데이터
예상 코드 리뷰 노력🎯 4 (복잡) | ⏱️ ~45분 관련 PR
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@debug/regression/questions/graduate_department_life.json`:
- Around line 1-2: This file's JSON uses per-case "category" fields inside the
"cases" array but lacks a top-level "category" like other files; either add a
top-level "category" key (e.g., "category": "graduate_department_life")
alongside "cases" to match the schema used by academic_schedule.json,
contacts.json, policies.json, or update the schema/validator to accept per-case
categories; locate the "cases" array in graduate_department_life.json and either
insert the top-level "category" entry or adjust the validation logic that
references "category" to accept case-level category keys.
- Around line 64-71: The test case with id "department_visual_design_intro" has
mismatched URL expectations: expected_url_contains is ["4647/subview.do"] but
answer_must_contain_any contains "4649/subview.do", which weakens the source URL
check when requires_source_url is true; fix by making the URLs consistent—either
change answer_must_contain_any to include "4647/subview.do" (or replace 4649
with 4647) or, if 4649 should also be acceptable, update expected_url_contains
and the source URL validation criteria to include both "4647/subview.do" and
"4649/subview.do" so source_url_match reflects the intended acceptance.
In `@README.md`:
- Around line 93-97: Update the README table row labeled "CI 경량 회귀": replace the
"synthetic + schema" description with a clear note that the CI job performs
schema-only validation using the --validate-only flag (no synthetic test cases
are executed), and replace the ambiguous "리포트 기준값" text with an explicit
baseline report location name (e.g., baseline_report.json or
rag_eval_report.json) so readers know where the baseline is stored; ensure the
changes specifically touch the "CI 경량 회귀" row and the columns that currently
show "synthetic + schema" and "리포트 기준값".
In `@tests/regression/evaluate_rag_retrieval.py`:
- Around line 178-204: The test currently collapses multiple failure sources
into a single "error" field (e.g., import_error, retrieval_error, answer_error)
which loses which stage failed; update the result objects created in the import
try/except and the other similar result constructions to preserve stage-specific
error fields (keep import_error, add retrieval_error and answer_error keys where
applicable) instead of merging them into one "error" string, and ensure any
later logic that sets retrieval_error or answer_error assigns into those
dedicated keys (e.g., the import block around importing
LLM.sub_model.query_index and the result dicts built after
schedule_search/hybrid_search) so both import and runtime errors are reported
independently; apply the same change to the other blocks mentioned (the result
constructions at the other ranges) so reports always expose import_error,
retrieval_error, and answer_error separately.
- Around line 65-95: validate_cases currently only checks presence of fields but
not their types; update validate_cases to validate types for
expected_url_contains and expected_title_contains (must be list[str] or
empty/absent), answer_must_contain_any and answer_must_contain_all (must be
list[str]), and boolean flags like requires_source_url (must be bool); for each
invalid type append an error like "{prefix}:invalid_type:<field>" (use the
existing prefix variable and case id logic), use isinstance checks and ensure
list items are str (reject non-list or lists with non-str items), and keep
existing presence checks (e.g., missing_expected_source) but only after type
validation to avoid treating strings as iterables.
- Line 391: The default output argument currently hardcodes a world-writable
/tmp path in the ap.add_argument("--out", ...) call which can cause collisions
and symlink attacks; change the default to a safer option (e.g., no default or a
path inside the repository such as a reports/ subdir) and when the CLI runs,
create a unique tempfile or use tempfile.NamedTemporaryFile/tmpdir or pathlib to
ensure a per-run unique and non-world-writable file; update the code that
consumes the --out value to create the directory if missing and write to that
resolved safe path instead of assuming /tmp.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: d95766e2-8801-409b-adf7-0e47af8eb300
📒 Files selected for processing (8)
.github/workflows/rag-light-check.yml.gitignoreREADME.mddebug/regression/questions/academic_schedule.jsondebug/regression/questions/contacts.jsondebug/regression/questions/graduate_department_life.jsondebug/regression/questions/policies.jsontests/regression/evaluate_rag_retrieval.py
Comment on lines
+1
to
+2
| { | ||
| "cases": [ |
There was a problem hiding this comment.
스키마 일관성: 최상위 category 필드가 누락되었습니다.
다른 질문 세트 파일(academic_schedule.json, contacts.json, policies.json)은 모두 최상위 레벨에 "category" 필드를 선언하지만, 이 파일은 각 케이스마다 개별 category 필드를 사용합니다. 스키마 검증 로직이 두 패턴을 모두 허용하는지 확인이 필요하며, 일관성을 위해 다른 파일과 동일한 구조를 사용하는 것을 권장합니다.
♻️ 제안된 수정 (일관성 개선)
만약 이 파일의 모든 케이스가 혼합된 카테고리를 가지므로 최상위 category가 적절하지 않다면, 현재 구조를 유지하되 스키마 검증 코드가 이를 지원하는지 확인하세요. 그렇지 않고 단일 카테고리로 그룹화 가능하다면:
{
+ "category": "mixed",
"cases": [
{
"id": "grad_professional_associate",
- "category": "grad",
"kind": "search",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@debug/regression/questions/graduate_department_life.json` around lines 1 - 2,
This file's JSON uses per-case "category" fields inside the "cases" array but
lacks a top-level "category" like other files; either add a top-level "category"
key (e.g., "category": "graduate_department_life") alongside "cases" to match
the schema used by academic_schedule.json, contacts.json, policies.json, or
update the schema/validator to accept per-case categories; locate the "cases"
array in graduate_department_life.json and either insert the top-level
"category" entry or adjust the validation logic that references "category" to
accept case-level category keys.
Comment on lines
+64
to
+71
| "id": "department_visual_design_intro", | ||
| "category": "department", | ||
| "kind": "search", | ||
| "query": "시각디자인과 소개", | ||
| "expected_url_contains": ["4647/subview.do"], | ||
| "expected_title_contains": ["시각디자인과"], | ||
| "answer_must_contain_any": ["시각디자인과", "4649/subview.do"], | ||
| "requires_source_url": true |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: Verify actual visual design department URLs in the search index
rg -i "시각디자인과" --type=py -C3 -g '!debug/**'Repository: dongsooop/AI
Length of output: 38
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== File snippet ==="
nl -ba debug/regression/questions/graduate_department_life.json | sed -n '50,90p'
echo "=== Find expected URL substrings ==="
rg -n --fixed-string "4647/subview.do" .
rg -n --fixed-string "4649/subview.do" .
echo "=== Find 시각디자인과 occurrences (limited to repo relevant dirs) ==="
rg -n -i "시각디자인과" debug . || trueRepository: dongsooop/AI
Length of output: 138
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== File snippet ==="
nl -ba debug/regression/questions/graduate_department_life.json | sed -n '50,90p'
echo "=== Find expected URL substrings ==="
rg -n --fixed-string "4647/subview.do" .
rg -n --fixed-string "4649/subview.do" .
echo "=== Find 시각디자인과 occurrences (limited to debug + rest) ==="
rg -n -i "시각디자인과" debug . || trueRepository: dongsooop/AI
Length of output: 138
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== File snippet ==="
nl -ba debug/regression/questions/graduate_department_life.json | sed -n '50,90p'
echo "=== Find expected URL substrings ==="
rg -n --fixed-string "4647/subview.do" .
rg -n --fixed-string "4649/subview.do" .
echo "=== Find 시각디자인과 occurrences (limited to debug + rest) ==="
rg -n -i "시각디자인과" debug . || trueRepository: dongsooop/AI
Length of output: 138
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="debug/regression/questions/graduate_department_life.json"
echo "=== File snippet (awk with line numbers) ==="
awk 'NR>=50 && NR<=90 {printf "%d:%s\n", NR, $0}' "$FILE"
echo "=== Find expected URL substrings ==="
rg -n --fixed-string "4647/subview.do" .
rg -n --fixed-string "4649/subview.do" .
echo "=== Find 시각디자인과 occurrences ==="
rg -n -i "시각디자인과" . || trueRepository: dongsooop/AI
Length of output: 2113
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="debug/regression/questions/graduate_department_life.json"
echo "=== File snippet (awk with line numbers) ==="
awk 'NR>=50 && NR<=90 {printf "%d:%s\n", NR, $0}' "$FILE"
echo "=== Find expected URL substrings (4647 / 4649) ==="
rg -n --fixed-strings "4647/subview.do" .
rg -n --fixed-strings "4649/subview.do" .
echo "=== Find '시각디자인과' occurrences ==="
rg -n -i "시각디자인과" . || trueRepository: dongsooop/AI
Length of output: 2716
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "expected_url_contains" .
rg -n "answer_must_contain_any" .Repository: dongsooop/AI
Length of output: 12396
🏁 Script executed:
#!/bin/bash
set -euo pipefail
FILE="tests/regression/evaluate_rag_retrieval.py"
echo "=== evaluate_rag_retrieval.py: top relevant section ==="
sed -n '60,120p' "$FILE"
echo "=== evaluate_rag_retrieval.py: expected_urls / answer checks ==="
sed -n '200,270p' "$FILE"Repository: dongsooop/AI
Length of output: 5687
테스트 기대 URL과 답변 키워드 URL이 불일치(의도 약화)
debug/regression/questions/graduate_department_life.json에서 expected_url_contains는 4647/subview.do를 기대하지만, requires_source_url: true일 때는 답변에 expected_url_contains(즉 4647/subview.do)가 포함됐는지로 source_url_match를 판단합니다. 그런데 answer_must_contain_any에 다른 URL인 4649/subview.do가 들어가 있어 답변에 4649가 없어도 "시각디자인과"만으로 통과할 수 있습니다(테스트 의도가 흐려짐). answer_must_contain_any를 4647/subview.do로 통일하거나, 4649를 함께 허용하려면 expected_url_contains/검증 기준도 같이 조정하는 편이 좋습니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@debug/regression/questions/graduate_department_life.json` around lines 64 -
71, The test case with id "department_visual_design_intro" has mismatched URL
expectations: expected_url_contains is ["4647/subview.do"] but
answer_must_contain_any contains "4649/subview.do", which weakens the source URL
check when requires_source_url is true; fix by making the URLs consistent—either
change answer_must_contain_any to include "4647/subview.do" (or replace 4649
with 4647) or, if 4649 should also be acceptable, update expected_url_contains
and the source URL validation criteria to include both "4647/subview.do" and
"4649/subview.do" so source_url_match reflects the intended acceptance.
Comment on lines
+93
to
+97
| | 구분 | 질문 세트 | 검색 Recall@3 | 답변 근거성 | 비고 | | ||
| | --- | ---: | ---: | ---: | --- | | ||
| | 개선 전 기준선 | 30개 | 리포트 기준값 | 리포트 기준값 | `/tmp/rag_eval_report.json` 또는 저장된 baseline 리포트 | | ||
| | 현재 브랜치 | 30개 | `summary.top3_url_accuracy` | `summary.source_url_pass_rate` | `evaluate_rag_retrieval.py` 실행 결과로 갱신 | | ||
| | CI 경량 회귀 | synthetic + schema | 통과/실패 | 통과/실패 | 무거운 임베딩 모델 다운로드 없이 PR에서 빠르게 검증 | |
There was a problem hiding this comment.
표의 "CI 경량 회귀" 행 설명을 명확히 하세요.
Line 97에서 "synthetic + schema"라고 기술했으나, 실제 워크플로우(.github/workflows/rag-light-check.yml)는 --validate-only로 스키마 검증만 수행합니다. "synthetic" 테스트 케이스는 현재 워크플로우에 포함되지 않았으므로 독자에게 혼란을 줄 수 있습니다.
또한 Line 95의 "리포트 기준값"이 어디에 저장되는지(예: baseline_report.json 파일 경로) 명시하면 더 명확합니다.
📝 제안된 수정
-| 개선 전 기준선 | 30개 | 리포트 기준값 | 리포트 기준값 | `/tmp/rag_eval_report.json` 또는 저장된 baseline 리포트 |
+| 개선 전 기준선 | 30개 | 리포트 기준값 | 리포트 기준값 | 저장된 baseline 리포트 파일 (예: `baseline_report.json`) |
| 현재 브랜치 | 30개 | `summary.top3_url_accuracy` | `summary.source_url_pass_rate` | `evaluate_rag_retrieval.py` 실행 결과로 갱신 |
-| CI 경량 회귀 | synthetic + schema | 통과/실패 | 통과/실패 | 무거운 임베딩 모델 다운로드 없이 PR에서 빠르게 검증 |
+| CI 경량 회귀 | schema | 통과/실패 | 통과/실패 | 스키마 검증 및 Python 컴파일 체크만 수행 (임베딩 모델 다운로드 없음) |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| | 구분 | 질문 세트 | 검색 Recall@3 | 답변 근거성 | 비고 | | |
| | --- | ---: | ---: | ---: | --- | | |
| | 개선 전 기준선 | 30개 | 리포트 기준값 | 리포트 기준값 | `/tmp/rag_eval_report.json` 또는 저장된 baseline 리포트 | | |
| | 현재 브랜치 | 30개 | `summary.top3_url_accuracy` | `summary.source_url_pass_rate` | `evaluate_rag_retrieval.py` 실행 결과로 갱신 | | |
| | CI 경량 회귀 | synthetic + schema | 통과/실패 | 통과/실패 | 무거운 임베딩 모델 다운로드 없이 PR에서 빠르게 검증 | | |
| | 구분 | 질문 세트 | 검색 Recall@3 | 답변 근거성 | 비고 | | |
| | --- | ---: | ---: | ---: | --- | | |
| | 개선 전 기준선 | 30개 | 리포트 기준값 | 리포트 기준값 | 저장된 baseline 리포트 파일 (예: `baseline_report.json`) | | |
| | 현재 브랜치 | 30개 | `summary.top3_url_accuracy` | `summary.source_url_pass_rate` | `evaluate_rag_retrieval.py` 실행 결과로 갱신 | | |
| | CI 경량 회귀 | schema | 통과/실패 | 통과/실패 | 스키마 검증 및 Python 컴파일 체크만 수행 (임베딩 모델 다운로드 없음) | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` around lines 93 - 97, Update the README table row labeled "CI 경량
회귀": replace the "synthetic + schema" description with a clear note that the CI
job performs schema-only validation using the --validate-only flag (no synthetic
test cases are executed), and replace the ambiguous "리포트 기준값" text with an
explicit baseline report location name (e.g., baseline_report.json or
rag_eval_report.json) so readers know where the baseline is stored; ensure the
changes specifically touch the "CI 경량 회귀" row and the columns that currently
show "synthetic + schema" and "리포트 기준값".
| ap.add_argument("--top-k", type=int, default=5, help="hybrid_search top_k for retrieval metrics") | ||
| ap.add_argument("--answer-top-k", type=int, default=6, help="build_answer top_k for answer metrics") | ||
| ap.add_argument("--schedule-top-k", type=int, default=5, help="schedule_search top_k for schedule cases") | ||
| ap.add_argument("--out", default="/tmp/rag_eval_report.json", help="output report path") |
There was a problem hiding this comment.
기본 출력 경로를 고정 /tmp 파일로 두지 않는 편이 좋습니다.
고정된 world-writable 경로는 병렬 실행 시 충돌하기 쉽고, 로컬에서는 사전 생성된 심볼릭 링크를 따라갈 위험도 있습니다. 기본값은 저장소 내부 경로나 실행마다 고유한 tempfile로 바꾸는 쪽이 안전합니다.
제안 diff
- ap.add_argument("--out", default="/tmp/rag_eval_report.json", help="output report path")
+ ap.add_argument(
+ "--out",
+ default=str(ROOT_DIR / "debug" / "regression" / "rag_eval_report.json"),
+ help="output report path",
+ )🧰 Tools
🪛 Ruff (0.15.13)
[error] 391-391: Probable insecure usage of temporary file or directory: "/tmp/rag_eval_report.json"
(S108)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/regression/evaluate_rag_retrieval.py` at line 391, The default output
argument currently hardcodes a world-writable /tmp path in the
ap.add_argument("--out", ...) call which can cause collisions and symlink
attacks; change the default to a safer option (e.g., no default or a path inside
the repository such as a reports/ subdir) and when the CLI runs, create a unique
tempfile or use tempfile.NamedTemporaryFile/tmpdir or pathlib to ensure a
per-run unique and non-world-writable file; update the code that consumes the
--out value to create the directory if missing and write to that resolved safe
path instead of assuming /tmp.
This was referenced May 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 배경
🔍 주요 내용
debug/regression/questions/*.json에 학사일정, 연락처, 장학금, 수강신청, 졸업, 학과/기숙사 질문 세트 30개를 추가했습니다.tests/regression/evaluate_rag_retrieval.py가 질문 디렉터리 로딩, schema 검증, top-k 검색 평가, 답변 키워드/출처/날짜/환각 proxy 리포트를 지원하도록 개선했습니다.RAG Light Check워크플로에서 질문 세트 schema와 query-index 경량 회귀를 검증하도록 연결했습니다.변경 요약(1~3줄)
RAG(검색-생성) 품질 평가 파이프라인을 추가해 회귀용 질문 세트(30개)로 검색 리콜·답변 근거성·환각 여부 등을 자동으로 검증하도록 했습니다. 평가 스크립트와 경량 CI 워크플로우를 보강하고 문서와 .gitignore를 업데이트했습니다.
주요 변경점
주의/리스크
다음 액션