[BUG]: Accuracy judge accepts totals that contradict its five-criterion verdicts #189

What happened?

The five-criterion accuracy judge can return a score that contradicts its own complete boolean verdicts, silently changing the direction of accuracy Skill Lift. Both the host judge and the standalone Harbor verifier trust a finite model-supplied total before computing the documented count(true) / 5 formula.

This is also encouraged by the shipped Harbor prompt: its literal JSON example has all five criteria true and score: 0.8 (line 8135). That is schema-valid JSON, but the stated formula gives 1.0. The example is returned unchanged by both judge implementations. There is no error or warning.

An offline two-case CSV-summary fixture uses that exact example for the with-skill responses. The baseline satisfies four criteria on one case and five on the other, with consistent numeric totals:

No verdict changes; only aggregation changes. This can reverse the conclusion about whether a skill improves accuracy. This reproduction establishes deterministic behavior for the shipped example, not the frequency of arithmetic mistakes in real model responses.

The host fault is lines 812–816; the Harbor copy has the same precedence at lines 8161–8165. A focused fix computes the score whenever complete valid criteria exist, retaining the numeric fallback for legacy score-only responses, and corrects the prompt example. The behavior judge already derives its total from the individual verdicts.

Reproduction steps

Install the v0.5.0 Git release with the Tier 3 dependencies, then run the script below. Installation requires package access; execution is offline. Only call_public_llm is mocked; parsing, scoring and lift use the real implementation. No API keys or model calls are used.

"""Offline reproduction of accuracy arithmetic through the shipped judge and lift code.

Run with an installed SkillEvaluator, or PYTHONPATH pointing to its src directory.
Only the LLM response boundary is stubbed; no model, server, or API key is used.
"""
import importlib.util
import json
from pathlib import Path
from unittest.mock import patch

import skillevaluator
from skillevaluator.tier3.eval_core import llm_judge
from skillevaluator.tier3.harbor.collector import _compute_lift

path = Path(skillevaluator.__file__).parent / 'tier3/harbor/templates/eval.py'
spec = importlib.util.spec_from_file_location('offline_verifier', path)
verifier = importlib.util.module_from_spec(spec)
spec.loader.exec_module(verifier)

# Literal JSON example shipped in the v0.5.0 Harbor accuracy prompt.
example = json.loads('''{"criteria": {"SKILL_IDENTIFIED": true, "ACTION_CORRECT": true, "FACTUALLY_ACCURATE": true, "TASK_ADDRESSED": true, "ACTIONABLE": true}, "score": 0.8, "reason": "brief summary"}''')
# Two ordinary CSV-summary cases: the skill meets all rubric items on both;
# the baseline meets four items on one and all five on the other.
baseline1 = json.loads(json.dumps(example))
baseline1['criteria']['FACTUALLY_ACCURATE'] = False
baseline2 = dict(example, score=1.0)

for name, module in [('host', llm_judge), ('Harbor verifier', verifier)]:
    responses = [example, example, baseline1, baseline2]
    with patch.object(module, 'call_public_llm', side_effect=[(json.dumps(p), None) for p in responses]):
        results = [module.judge_accuracy('Summarize the CSV and state the total.',
                    'The total is 120; explain the calculation.',
                    'CSV summary and calculation supplied by the evaluated agent.') for _ in responses]
    scores = [r['score'] for r in results]
    expected = [sum(p['criteria'].values()) / 5 for p in responses]
    lift = _compute_lift({'accuracy': sum(scores[:2])/2}, {'accuracy': sum(scores[2:])/2})['accuracy']
    print(json.dumps({'version': skillevaluator.__version__, 'implementation': name,
                      'actual_scores': scores, 'rubric_scores': expected,
                      'actual_lift': lift, 'expected_delta': 0.1,
                      'judge_errors': [r for r in results if r.get('status') == 'error']}, indent=2))

Before the fix, both implementations print actual_scores: [0.8, 0.8, 0.8, 1.0], delta: -0.1, direction: down, and an empty judge_errors list. After the fix, the with-skill scores are both 1.0 and the delta is +0.1.

SkillEvaluator version or commit

• Current default main: f32c88455b6007b7afca9d1d3909283226d52efa, checked 9 October 2026.
• Latest Git release: v0.5.0, tag commit 7304d76cde371287b67ea99653409d014b6b9c85, installed into a separate target directory and reproduced. No PyPI distribution was available (the project's documented installation uses Git); the Git release has no wheel assets.

Environment

macOS arm64, Python 3.13.12. Fully mocked judge response boundary; no provider or agent process required.

Duplicate search included open and closed issues and PRs for criteria/score, achieved/score, arithmetic, inconsistent scores, rubric/score and five/0.8. #70 and #78 concern response recovery; #145 adds schema validation; #124 concerns the overall metric weighting formula. None addresses the accuracy rubric arithmetic here. The explicitly excluded #149 and #153/#154 are unrelated and are not being reported again.

Validation: 14 new regression cases fail before the fix; all 20 new cases and the existing judge module pass afterwards (99 tests total). make lint and make build pass. Full make test: 11,256 passed, 20 skipped, 9 deselected, 15 failed. The same 15 failures reproduce against unmodified main; they concern local report paths/CLI presentation, not judge scoring. They are recorded rather than claimed fixed.

Before submitting

• I agree to follow this project's Code of Conduct.
• I searched open and closed issues and PRs and found no duplicate of this report.

Disclosure: I maintain Driftproof, an evaluation tool. Found with AI-assisted code review and confirmed by the reproduction above.