Skip to content
View shryu1994's full-sized avatar

Block or report shryu1994

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
shryu1994/README.md

Cloud/SRE engineer in regulated industries — I build AI that refuses rather than guesses.

Public sector → finance → medical devices, 4.5 years. Today I'm the sole owner of one company's GCP estate: security governance, infrastructure, and the AI running on top of it. Working on compliance documentation taught me that a confident wrong answer isn't a demo bug — it's a compliance event. So I started building the measurement I needed and couldn't find: benchmarks where a correct "I don't know" is a first-class pass.

규제 산업(공공·금융·의료기기)의 클라우드를 4년 6개월 운영해 왔고, 지금은 한 회사의 GCP 전체를 혼자 맡고 있습니다. 규제 문서를 다루다 보니 AI의 "그럴듯한 오답"은 데모 버그가 아니라 컴플라이언스 사고라는 걸 알게 됐고, 그래서 "모른다"고 답하는 것도 성능으로 채점하는 평가 도구를 만들어 공개합니다.


Start here · 여기부터

provenance-bench — a faithfulness + justified-abstention benchmark for regulated documentation. The conventional joint score rewards a naive "always answer" baseline — it wins at 0.70 while doing the right thing on only 29.2% of items. So the headline here is an un-gameable overall_correct instead: 0.292 → 0.667. On the 46 cases that require a refusal, naive abstains 0 times; the grounded method catches 42 (recall 0.913). make demo reproduces it offline. (v0.2 · a 72-case synthetic seed set — small on purpose; the method is the point, and it cites the prior work it stands on.)
› 관행 지표로 채점하면 "무조건 답하기"가 이깁니다. 속일 수 없는 지표로 다시 설계했습니다.

cite-or-refuse — the grounded RAG method behind that number: cite a real source, or honestly refuse. Refusal is scored as a first-class eval, with a faithfulness judge. Fully offline, synthetic data.
› 근거가 없으면 답하지 않는 검색-답변(RAG) 시스템.

promptprint — 100% local, zero-dependency analysis of the prompts you send your AI coding agents. See how your questioning grew over time.
› AI에게 던진 질문이 어떻게 늘었는지 로컬에서 분석.


Stack

Cloud Kubernetes (CKA · CKAD) · GCP (GKE · Cloud Run · BigQuery) · Terraform · ArgoCD (GitOps) · Docker · Helm · NCP (Professional) · KT Cloud AI Python · FastAPI · RAG (Qdrant · Neo4j) · MCP servers · Vertex AI · LLM output verification Ops & Security Prometheus · Grafana · cloud security governance (CSPM · IAM) · WAF · incident response · FinOps

More at sh-ryu.com · LinkedIn

Pinned Loading

  1. provenance-bench provenance-bench Public

    A faithfulness + justified-abstention benchmark for regulated documentation — a correct refusal is a first-class pass, credited only for the right reason.

    Python

  2. cite-or-refuse cite-or-refuse Public

    A grounded RAG QA system that cites its sources or honestly refuses — refusal scored as a first-class eval. Fully offline, synthetic data.

    Python

  3. promptprint promptprint Public

    See how your questioning skill grew up — a 100% local, zero-dependency analysis of the prompts you send your AI coding agents.

    Python