Skip to content

feat(vera_mh): add VERA-MH clinical-rubric safety benchmark #3620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
reinainblood wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(vera_mh): add VERA-MH clinical-rubric safety benchmark #3620

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
40dac33
feat(vera_mh): add VERA-MH clinical-rubric safety benchmark
reinainblood Sep 22, 2026
86a2e58
Merge upstream main into vera-mh-upstream-clean
reinainblood Oct 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/workflows/config/.secrets.baseline
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions ATTRIBUTIONS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -218,13 +218,15 @@ upstream copyright header and adds an NVIDIA modifications block.
| tau2-bench | MIT (original); Apache-2.0 (NVIDIA modifications) | `resources_servers/indian_banking/core/` (`action_compare.py`, `state_normalize.py`; adapted judge prompt shape in `judge.py`, edited user-simulator guidelines in `prompts/user-sim-guidelines.md`) | https://github.com/sierra-research/tau2-bench |
| AssayBench reference harness | MIT (original); Apache-2.0 (NVIDIA modifications) | `resources_servers/assaybench/gene_parsing.py` (`parse_genes_from_output`, `extract_genes_from_raw_response` from `benchmarking/predictions_generation/collect_llm_predictions.py`; formatting and type hints only) | https://github.com/Genentech/AssayBench |
| ReactionMechanismReasoning (FukuyamaBench) | Apache-2.0 (original and NVIDIA modifications) | `resources_servers/fukuyamabench/` (pathway scorer ported into `metrics.py` from upstream `eval/eval_infer_pathway.py`; system prompt and user template copied into `scripts/prepare_fukuyamabench.py` from upstream `eval/prompts/infer_pathway_prompts.yaml` — the user template byte-identical, the system prompt content-preserving with trailing whitespace stripped) | https://github.com/HaCTang/ReactionMechanismReasoning |
| VERA-MH | Spring Care VERA-MH license (original; MIT-style permission notice plus disclaimer); Apache-2.0 (NVIDIA modifications) | `resources_servers/vera_mh/rubric/` (`rubric.tsv`, `rubric_prompt_beginning.txt`, `question_prompt.txt`, `persona_prompt_template.txt`, `persona_prompt_reminder.txt` copied byte-identical from upstream `data/`, with `LICENSE`); `resources_servers/vera_mh/app.py` (judge flow, navigation and scoring ported from upstream `judge/llm_judge.py`, `judge/question_navigator.py`, `judge/rubric_config.py`, `judge/score_utils.py`, `judge/score.py`); `responses_api_agents/vera_mh_agent/app.py` (conversation loop and message construction ported from `generate_conversations/conversation_simulator.py` and `utils/conversation_utils.py`) | https://github.com/SpringCare/VERA-MH |

- Upstream license: reproduced in full in the Apple MIT License section below
- Subcomponent notices: `resources_servers/toolsandbox/tool_sandbox/ACKNOWLEDGEMENTS`
- Provenance & list of NVIDIA modifications: `resources_servers/toolsandbox/tool_sandbox/VENDORING.md`
- tau2-bench: original MIT notices preserved in each vendored file; NVIDIA modifications noted in-file and in `resources_servers/indian_banking/README.md`
- AssayBench: original MIT notice preserved in the vendored file; the `assaybench` metric package itself is a pinned PyPI dependency of that server (`resources_servers/assaybench/requirements.txt`), not vendored
- ReactionMechanismReasoning: pinned at upstream revision `63bb79f912f0b2de593996b80ffeea894f6f1a59`. Upstream is Apache-2.0, the same license as this repository, so no additional license text is required. NVIDIA modifications to the ported scorer are documented at their call sites in `metrics.py` and summarised in `resources_servers/fukuyamabench/README.md`. Benchmark data is **not** vendored; it is downloaded at runtime by `scripts/prepare_fukuyamabench.py`.
- VERA-MH: pinned at upstream revision `2c9d1fcbb68e1a2df64171c18b3e4d4c18b2f89e` (rubric v1.2). The upstream license is reproduced in full at `resources_servers/vera_mh/rubric/LICENSE`, beside the vendored files, with per-file hashes in `resources_servers/vera_mh/rubric/PROVENANCE.md`; it permits use and redistribution with the notice intact and states that a VERA-MH score is not a certification, endorsement or safety determination. The ported judge and simulator code cites its upstream source per function. The persona sheet is **not** vendored; `benchmarks/vera_mh/prepare.py` downloads it from the pinned revision and verifies its hash.

---

Expand Down
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,6 +408,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
| Toolsandbox | agent | ToolSandbox multi-turn tool-use benchmark; reward is the milestone similarity in [0, 1] (0 if a minefield is hit). | Improve stateful, multi-turn tool-calling and user-interaction capability. | - | ✓ | - | <a href='resources_servers/toolsandbox/configs/toolsandbox.yaml'>toolsandbox.yaml</a> | - |
| Ugphysics Judge | knowledge | Undergraduate physics QA verified by a TRUE/FALSE LLM judge with math-verify symbolic fallback | Score undergraduate-physics benchmarks (e.g. UGPhysics) where the judge is a TRUE/FALSE equivalence grader using a reference solution | - | - | - | <a href='resources_servers/ugphysics_judge/configs/ugphysics_judge.yaml'>ugphysics_judge.yaml</a> | - |
| Vcqa Agent | coding | Verified Code QA - investigate a per-task repo snapshot or git bundle, then answer against a must-have rubric graded by an LLM judge. | Code investigation across fileset and git-history tasks. | ✓ | ✓ | Apache 2.0 | <a href='responses_api_agents/vcqa_agent/configs/vcqa_agent.yaml'>vcqa_agent.yaml</a> | <a href='https://huggingface.co/datasets/appliedcompute/vcqa-v1'>vcqa-v1</a> |
| Vera Mh | safety | VERA-MH simulated mental-health crisis conversations judged with the clinical v1.2 rubric by GPT 5.4 | Measure suicide-risk detection, confirmation, guidance to human care, supportive conversation and AI boundaries in multi-turn chats | - | - | - | <a href='resources_servers/vera_mh/configs/vera_mh.yaml'>vera_mh.yaml</a> | - |
| Verifiers Agent | math | Prime intellect verifiers and environments hub integration, ace-reason math environment example. | Improve math reasoning capabilities. | ✓ | - | - | <a href='responses_api_agents/verifiers_agent/configs/verifiers_agent.yaml'>verifiers_agent.yaml</a> | - |
| Verifif | instruction_following | VerifIF instruction following validators with rule-based and LLM judge support | Improve instruction following capabilities with comprehensive validation | - | - | - | <a href='resources_servers/verifif/configs/verifif.yaml'>verifif.yaml</a> | - |
| Vibench | coding | PRD-to-web-app benchmark; apps are graded by seeding and driving them in a browser. | Eval whether a model can ship a working web app from a spec, judged through the running UI rather than the diff. | - | - | - | <a href='resources_servers/vibench/configs/vibench.yaml'>vibench.yaml</a> | - |
Expand Down
Loading