Public sector → finance → medical devices, 4.5 years. Today I'm the sole owner of one company's GCP estate: security governance, infrastructure, and the AI running on top of it. Working on compliance documentation taught me that a confident wrong answer isn't a demo bug — it's a compliance event. So I started building the measurement I needed and couldn't find: benchmarks where a correct "I don't know" is a first-class pass.
규제 산업(공공·금융·의료기기)의 클라우드를 4년 6개월 운영해 왔고, 지금은 한 회사의 GCP 전체를 혼자 맡고 있습니다. 규제 문서를 다루다 보니 AI의 "그럴듯한 오답"은 데모 버그가 아니라 컴플라이언스 사고라는 걸 알게 됐고, 그래서 "모른다"고 답하는 것도 성능으로 채점하는 평가 도구를 만들어 공개합니다.
provenance-bench — a faithfulness +
justified-abstention benchmark for regulated documentation.
The conventional joint score rewards a naive "always answer" baseline — it wins at 0.70
while doing the right thing on only 29.2% of items. So the headline here is an un-gameable
overall_correct instead: 0.292 → 0.667. On the 46 cases that require a refusal, naive
abstains 0 times; the grounded method catches 42 (recall 0.913). make demo
reproduces it offline. (v0.2 · a 72-case synthetic seed set — small on purpose; the method is
the point, and it cites the prior work it stands on.)
› 관행 지표로 채점하면 "무조건 답하기"가 이깁니다. 속일 수 없는 지표로 다시 설계했습니다.
cite-or-refuse — the grounded RAG method
behind that number: cite a real source, or honestly refuse. Refusal is scored as a first-class
eval, with a faithfulness judge. Fully offline, synthetic data.
› 근거가 없으면 답하지 않는 검색-답변(RAG) 시스템.
promptprint — 100% local, zero-dependency
analysis of the prompts you send your AI coding agents. See how your questioning grew over time.
› AI에게 던진 질문이 어떻게 늘었는지 로컬에서 분석.
Cloud Kubernetes (CKA · CKAD) · GCP (GKE · Cloud Run · BigQuery) · Terraform · ArgoCD (GitOps) · Docker · Helm · NCP (Professional) · KT Cloud AI Python · FastAPI · RAG (Qdrant · Neo4j) · MCP servers · Vertex AI · LLM output verification Ops & Security Prometheus · Grafana · cloud security governance (CSPM · IAM) · WAF · incident response · FinOps
More at sh-ryu.com · LinkedIn
