feat: add metadata based retrieval reranking - #153
Merged
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthrough연락처 및 조직 단위 질의의 정규화 규칙을 변경했습니다. 문서 메타데이터 기반 검색 점수 보정을 추가했습니다. 관련 회귀 테스트와 RAG 평가 사례를 확장했습니다. Changes검색 재순위 조정
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant hybrid_search
participant _metadata_retrieval_bonus
participant 색인_메타데이터
hybrid_search->>_metadata_retrieval_bonus: 질의와 문서 전달
_metadata_retrieval_bonus->>색인_메타데이터: 조직·출처·제목·breadcrumb 확인
색인_메타데이터-->>_metadata_retrieval_bonus: 메타데이터 반환
_metadata_retrieval_bonus-->>hybrid_search: 문서별 보너스 반환
hybrid_search->>hybrid_search: 최종 검색 점수 계산
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
LLM/sub_model/query_index.py (1)
570-608: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
search_df.iterrows()중복 및 성능 개선이 필요합니다.570-577번째 줄과 593-600번째 줄은 동일한 단위 매칭 로직(
unit,title,leaf_title에 대해_unit_term_score최댓값 계산)을iterrows()로 중복 구현합니다.iterrows()는 매 행마다 Series를 생성하므로 느립니다. 이 파일의 다른 부분(614-632번째 줄)은 이미.map()을 사용한 벡터화 패턴을 채택하고 있어 일관성이 떨어집니다.이 함수는 연락처 키워드가 포함된 질의마다 호출되며, 연락처 질의는 이 챗봇의 핵심 사용 사례입니다. 색인 규모가 커지면 요청 경로에서 지연 시간이 늘어날 수 있습니다.
공통 헬퍼로 추출하고
.apply()또는.map()기반으로 바꾸는 것을 권장합니다.♻️ 제안하는 리팩터링 방향
+def _unit_match_scores(target_unit: str, df: pd.DataFrame) -> np.ndarray: + """unit/title/leaf_title 열에서 target_unit과의 최대 매칭 점수(0~1)를 계산합니다.""" + def _row_score(row): + candidates = (row.get("unit", ""), row.get("title", ""), row.get("leaf_title", "")) + return max((_unit_term_score(target_unit, str(v)) for v in candidates), default=0) + return (df.apply(_row_score, axis=1) / 6.0).to_numpy()두 분기 모두 이 헬퍼를 호출하도록 정리하면 중복이 사라지고 유지보수가 쉬워집니다.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@LLM/sub_model/query_index.py` around lines 570 - 608, Extract the duplicated unit-score calculation into a shared helper near the surrounding ranking logic, computing the maximum _unit_term_score across unit, title, and leaf_title for one row. Replace both search_df.iterrows() loops that build unit_scores with a .apply() or .map()-based Series calculation using that helper, while preserving the existing normalization, canonical-page checks, contact-source matching, and bonus behavior in each branch.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@LLM/sub_model/query_index.py`:
- Around line 112-130: Update _known_unit_alias so its cached result cannot
outlive the underlying search_df index: either remove `@lru_cache` and evaluate
against the current index on every call, or include a stable search_df
identity/version in the cache key. Preserve the existing alias-matching logic
and empty-index behavior.
- Around line 561-635: Restructure the branching in the metadata bonus logic so
contact_like queries no longer return before the shared leaf-title and
query-term coverage calculations. Keep contact-specific handling and the
target_unit-dependent general handling mutually exclusive, preserve the contact
path’s existing scoring behavior without NON_CONTACT_DOC_PENALTY, and let all
queries reach the common EXACT_LEAF_RETRIEVAL_BOOST and
QUERY_TERM_COVERAGE_BOOST logic before returning bonus.
---
Nitpick comments:
In `@LLM/sub_model/query_index.py`:
- Around line 570-608: Extract the duplicated unit-score calculation into a
shared helper near the surrounding ranking logic, computing the maximum
_unit_term_score across unit, title, and leaf_title for one row. Replace both
search_df.iterrows() loops that build unit_scores with a .apply() or
.map()-based Series calculation using that helper, while preserving the existing
normalization, canonical-page checks, contact-source matching, and bonus
behavior in each branch.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 6da8c8f4-27ac-4cc9-b0eb-44ca469bb8da
📒 Files selected for processing (4)
LLM/sub_model/query_index.pytests/regression/chatbot/chatbot_regression_cases.jsontests/regression/chatbot/check_query_index_metadata_quality.pytests/regression/chatbot/rag_eval_cases.json
Comment on lines
+112
to
+130
| @lru_cache(maxsize=256) | ||
| def _known_unit_alias(candidate: str) -> bool: | ||
| """Accept a bare '-과' abbreviation only when it matches indexed unit metadata.""" | ||
| if not candidate.endswith("과") or len(_unit_root(candidate)) < 2: | ||
| return False | ||
|
|
||
| indexed_df = globals().get("search_df") | ||
| if indexed_df is None or indexed_df.empty: | ||
| return False | ||
|
|
||
| rows = indexed_df[indexed_df["doc_type"].isin(["contact", "department"])] | ||
| for column in ("unit", "leaf_title", "title"): | ||
| if column not in rows.columns: | ||
| continue | ||
| if rows[column].fillna("").astype(str).map( | ||
| lambda value: _unit_term_score(candidate, value) >= 4 | ||
| ).any(): | ||
| return True | ||
| return False |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Description: query_index.py 내에서 색인(search_df)을 런타임에 재적재하는 코드가 있는지 확인합니다.
set -euo pipefail
fd query_index.py LLM/sub_model --exec cat -n {}
rg -n 'search_df\s*=' LLM/sub_model/query_index.py
rg -n 'cache_clear|reload|_index_resources' LLM/sub_model/query_index.pyRepository: dongsooop/AI
Length of output: 43793
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== candidate files =="
fd -a '.*\.py$' LLM/sub_model | sed 's#^\./##'
echo "== query_index_loader.py =="
cat -n LLM/sub_model/query_index_loader.py
echo "== search_df/cache_clear/reload usages outside query_index.py =="
rg -n 'search_df\s*=|load_query_index_resources|query_index_loader|cache_clear|reload' LLM . --glob '*.py' | head -n 200Repository: dongsooop/AI
Length of output: 8771
_known_unit_alias를 캐시하지 않거나 캐시 키에 색인 식별자를 포함하세요.
_known_unit_alias는 candidate만으로 캐시하며 globals().get("search_df")를 기준으로 판정합니다. 색인 파일 기반의 search_df만 모듈 시작 시 로드하므로 런타임 내 재적재 로직은 현재 없지만, 색인 교체 후 같은 모듈을 사용하는 동시 요청에서는 이전 search_df의 결과를 계속 반환할 수 있습니다. 캐시는 제거하고 매번 색단을 평가하거나 search_df 식별자/버전을 캐시 키에 포함하세요.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@LLM/sub_model/query_index.py` around lines 112 - 130, Update
_known_unit_alias so its cached result cannot outlive the underlying search_df
index: either remove `@lru_cache` and evaluate against the current index on every
call, or include a stable search_df identity/version in the cache key. Preserve
the existing alias-matching logic and empty-index behavior.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/regression/chatbot/check_query_index_metadata_quality.py`:
- Around line 402-405: 테스트의 `_metadata_retrieval_bonus` 검증을 전체 배열의 `np.any` 확인에서
대상 행의 메타데이터 필터링으로 변경하세요. 일반 질의는 기대 URL과 `doc_type == "contact"`를 함께 만족하는 행의 보너스를
검증하고, 특정 질의는 여기에 `unit == "학생성공지원팀"`을 추가하세요. URL이 반복되는 경우에도 오답 조직 청크가 통과하지 않도록
하며, 재순위 의도라면 오답 조직 행과 비교해 대상 행의 보너스가 더 높은지도 확인하도록 `student_row` 및 관련 검증을 수정하세요.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 4cdba12d-9fce-4bfb-8441-3f2a0756f57a
📒 Files selected for processing (2)
LLM/sub_model/query_index.pytests/regression/chatbot/check_query_index_metadata_quality.py
🚧 Files skipped from review as they are similar to previous changes (1)
- LLM/sub_model/query_index.py
Yu-JeSeung
added
run-rag-check
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 배경
🔍 주요 내용
변경 요약
하이브리드 검색에 메타데이터 기반 재정렬을 추가했습니다. 부서 연락처와 부서 관련 문서의 검색 정확도를 개선합니다.
주요 변경점
주의/리스크
다음 액션