Repository navigation
MedReason #31
Description
Implement verifiers env for MedReason
Implement a verifiers environment for the MedReason dataset. Use the authors’ question, system, and LLM-as-judge prompts if provided. Only create a training dataset if a training split exists.
Paper: https://arxiv.org/abs/2504.00993
Dataset / Project: https://huggingface.co/datasets/UCSC-VLAA/MedReason
Format & Env
- Mixed (some multiple-choice, some open-ended):
- MC tasks: accuracy-based evaluation
- Open-ended reasoning:
vf.JudgeRubric()
Metadata
Metadata
Assignees
Labels
No labels
Activity
tmabraham commentedon Oct 7, 2025
Correct me if I am wrong, but my understanding is that MedReason is mainly a combination of existing datasets like PubMedQA, MedQA, etc. except that they are using some specialized knowledge graph-based approach for generating CoT traces which they can use for SFT. But this is not relevant for RL and evals I think, so for that it's just the same as existing datasets no?
warner-benjamin commentedon Oct 7, 2025
I was thinking the knowledge graph-based reasoning could be used for a LLM as a Judge reward, potentially making this dataset more useful than just MCQ datasets.
Manishram-ai commentedon Oct 8, 2025
I can try to take on this task :)
AdepojuJeremy commentedon Jan 21, 2026
Can you reassign this task to me
feat: add MedReason environment and environment validation tests
feat: add MedReason environment and environment validation tests