PubHealthBench freeform: inference and judge prompts are misaligned #139

Problem: In environments/pubhealthbench/pubhealthbench/prompts.py, the inference prompt asks "Please answer the question to the best of your knowledge", but the judge requires that "any additional detail in the given answer must be seen in the provided context." The judge sees only a few guidance chunks, so correct answers that include standard background detail fail, even when the judge itself says the core answer is right. This mainly penalises verbose chat and reasoning models.

Observation We used an RL-tuned Qwen3.5-4B with the same sampling throughout. Only the inference prompt changes; the concise prompt asks the model to answer only with what the guidance states.

Concise prompt

The following is a question about UK Government public health guidance.

Question: This question relates to UK Health Security Agency (UKHSA) guidance.
{question}

Please answer the question concisely, in one or two sentences, stating only what the UKHSA guidance says. Do not add background, extra steps, examples, or caveats.
Answer:

Providing more detailed inference prompt improves accuracy.

Example 1 (question 441). What must be done before accurate measurement of internal radiocaesium contamination? Ground truth: Removal of external contamination

Default answer (snippet), score 0: "The critical step… is the thorough decontamination of the person's skin and external clothing… Skin must be washed thoroughly with soap and water… Any remaining external contamination must be checked… using survey meters."

• gpt-oss-120b: "does state that removal of external contamination… is required, which aligns with the ground truth. However, it adds extensive additional details… not present in the provided context."
• gemma4-31b: "correctly identifies the core requirement… [but] since… 'any additional detail… must be seen in the provided context,' the answer is incorrect."
• GLM-5.3: "The core of the given answer does align with the ground truth… the answer cannot be considered correct despite capturing the core point."
• DeepSeek-V4.1-Flash: "correctly identifies removal of external contamination as the critical step. However, it adds substantial unsupported detail."

Concise prompt answer, score 1 (all four judges): "External contamination must be removed before accurate measurements of internal radiocaesium contamination can be performed."

Example 2 (question 684). What is the primary health effect of nitrobenzene exposure? Ground truth: Methaemoglobinaemia, which reduces the blood's ability to transport oxygen

Default answer (snippet), score 0: "The primary acute health effect… is methemoglobinemia… Nitrobenzene oxidizes the iron in hemoglobin from Fe²⁺ to Fe³⁺… Methemoglobin cannot bind and transport oxygen effectively… requiring prompt treatment with oxygen and… methylene blue."

• gpt-oss-120b: "correctly identifies the primary health effect as methemoglobinemia… matching the ground truth. However, it adds… the specific biochemical mechanism… not present in the supplied context."
• gemma4-31b: "correctly identifies the primary health effect… [but] includes… specific chemical mechanisms (Fe2+ to Fe3+)… specific treatments like methylene blue… the answer is incorrect."
• GLM-5.3: "correctly identifies the core fact… despite the core answer being correct, the given answer cannot be judged correct under the strict criteria."
• DeepSeek-V4.1-Flash: "correctly identifies methaemoglobinaemia… However, it includes extensive additional details not present in the provided context."

Concise prompt answer, score 1 (all four judges): "The primary health effect of exposure to nitrobenzene is methemoglobinemia, which impairs blood oxygen-carrying capacity."

Suggested fix (either option).

1. Align the inference prompt with the judge: "Answer with what the UKHSA guidance states. Include every point the guidance gives that answers the question, and nothing else. Be concise."
2. Relax the judge: fail only contradictions and omissions of key points, and report unsupported extra detail separately.