refactor(judge): unify LLM judge scoring on a single JSON output contract

.gitignore

@@ -168,6 +168,7 @@ examples/api_test/
 .claude
 .kiro
 .pr-review/
+.scratch_wl/
 
 # Frontend (evalscope/web)
 node_modules/

AGENTS.md

@@ -126,6 +126,21 @@ Don't try to learn the architecture from this file — read these and grep:
 6. Auto-discovered by globbing `evalscope/benchmarks/*/**/*_adapter.py`.
 7. Add a smoke test.
 
+## Adding a judge-scored benchmark
+
+An adapter must **never** call `self.llm_judge.judge()` or parse a judge reply itself — that debt is fenced off by `tests/api/judge/test_gates.py`, which scans every file under `evalscope/benchmarks/` (helpers included, so moving a parser into `utils.py` does not evade it). Score through the JSON output contract in `evalscope/api/judge/` instead:
+
+1. Pick a `scoring_policy` (`JUDGE_ONLY` or `JUDGE_DEFAULT`). Judge scoring always goes through the contract; there is no opt-in flag and no legacy path.
+2. **Single verdict per sample:** implement `judge_definition(context)` and return `JudgeDefinition.labels(...)` for a label mapping or `JudgeDefinition.numeric(...)` for a 0-1 rating. A generic `prompt_template` must state grading criteria only: `OutputContract.instruction()` appends the reply format. An adapter that preserves an official fixed output template may keep that format instruction instead, provided its `OutputContract` schema matches the official template.
+3. **Custom shape (multiple cases, ratings, rubrics):** `judge_definition(context)` declares a Pydantic `schema_model`, wraps it in `OutputContract`, and returns `JudgeDefinition.workflow(...)`:
+   - `cases` contains one `JudgeCase(case_id, output_contract, metadata)` per thing to judge.
+   - `request(case, placement, completed, context)` renders messages and appends `case.output_contract.instruction()` so the prompt and parser cannot drift. An official fixed output template may be used instead when its required fields and constraints match the case's `OutputContract` schema.
+   - `reduce(verdicts, context)` folds parsed verdicts into `{metric: value}`. Read a verdict's context from `CaseVerdict.metadata`, never by parsing `case_id`.
+   - Optional `expand`, `fallback`, and `finalize` callbacks handle staged cases, rule fallbacks, and score finalization. They may be nested functions or private adapter helpers, but are passed only through the returned definition.
+4. **Rule short-circuit:** if deterministic scoring settles the sample before judge I/O, return `JudgeDefinition.skip(score, reason='...')`. The non-empty reason is persisted in `Score.metadata` as `judge_skipped=True` and `judge_skip_reason`; the web review panel displays it as rule-based scoring.
+5. The executor owns request execution, position swap, repeats, multi-judge aggregation and fail-closed exclusion. Transport retries belong to the model implementation; a reply that fails the contract is not automatically retried and excludes the sample from the metric — never scored 0 or full credit — so a metric's `num` can be below the sample count.
+6. Add a scripted-judge test in `tests/api/judge/test_migrated_adapters.py` covering: a valid verdict, a parse failure (prose / malformed), and a transport `[ERROR]` — each must exclude, not silently score. A judge double must carry the surface the definition reads (`score_type`, `score_mapping`, `build_prompt`), and be injected through the `llm_judge` setter rather than a private attribute.
+
 ## Conventions & gotchas
 
 - `eval_type`: `openai_api`, `llm_ckpt`, `mock_llm`, `text2image`, `image_editing`. Deprecated aliases: `server` → `openai_api`, `checkpoint` → `llm_ckpt`.

docs/en/advanced_guides/add_benchmark.md

@@ -47,7 +47,7 @@ DataAdapter adopts a Pipeline architecture, supporting custom behavior through h
    calculate_metrics()
    ├── filter_prediction()
    │   └── extract_answer() [Optional User Implementation]
-   ├── match_score() / llm_match_score()
+   ├── match_score() / score_with_judge_contracts()
    └── Returns SampleScore
 
 4. Result Aggregation Phase
@@ -61,6 +61,31 @@ DataAdapter adopts a Pipeline architecture, supporting custom behavior through h
        └── Returns Report
 ```
 
+### Adding an LLM-Judged Benchmark
+
+Users enable judging with the typed `judge` configuration described in [Judge Parameters](../get_started/parameters.md#judge-parameters):
+
+```python
+TaskConfig(
+    model='MODEL_UNDER_TEST',
+    datasets=['your_benchmark'],
+    judge={
+        'strategy': 'llm',
+        'models': {'model_id': 'JUDGE_MODEL', 'api_url': 'OPENAI_COMPATIBLE_URL', 'api_key': 'API_KEY'},
+    },
+)
+```
+
+For an adapter author, judge I/O belongs to `evalscope.api.judge`; do not call `self.llm_judge.judge()` or parse a model reply in the adapter.
+
+1. Declare `scoring_policy`: use `JUDGE_ONLY` when rule scoring is not meaningful, `JUDGE_DEFAULT` when rules remain available but `auto` should judge, and `RULE_DEFAULT` when `auto` should keep rule scoring.
+2. Implement the single adapter entry point, `judge_definition(context)`. For an ordinary one-verdict task, return `JudgeDefinition.labels(...)` or `JudgeDefinition.numeric(...)` with a Pydantic verdict schema. These helpers append the JSON output instruction and keep the prompt, schema, and metric mapping together.
+3. For a rubric, multiple claims, or staged task, define a Pydantic verdict schema and `OutputContract`, then return `JudgeDefinition.workflow(cases=..., request=..., reduce=...)`. Pass `expand=...`, `fallback=...`, or `finalize=...` only when the workflow needs them. The callbacks can be nested in `judge_definition()` or private adapter helpers, but they must be owned by the returned definition rather than exposed as adapter hooks. Append `case.output_contract.instruction()` in a custom request unless the official fixed JSON instruction exactly matches that schema. Use `CaseVerdict.metadata` rather than encoding state into `case_id`.
+4. When a deterministic rule settles a sample without a model call, return `JudgeDefinition.skip(score, reason='...')`. `reason` is required and is persisted as `Score.metadata['judge_skip_reason']` with `Score.metadata['judge_skipped'] = True`; the web review panel labels this as rule-based scoring rather than an LLM verdict.
+5. Add a scripted-judge test covering a valid JSON verdict, malformed/prose output, and a transport error. Invalid judge replies are excluded from the metric; they are not converted to a zero score and are not automatically retried by the executor.
+
+The executor owns request dispatch, position swaps, repeats, multi-judge quorum, aggregation, and review diagnostics. The model transport owns its own retry policy through `generation_config`.
+
 ### Core Data Structures
 
 #### 1. Sample Object

docs/en/advanced_guides/custom_dataset/llm.md

@@ -240,14 +240,20 @@ run_task(task_cfg=task_cfg)
 
 **Method 2: Evaluation based on LLM**
 
-LLM-based evaluation can conveniently assess the correctness of model outputs (or other dimensions of metrics, requiring custom prompt settings). Below is an example configuring `judge_model_args` parameters, using the preset `pattern` mode to determine the correctness of model outputs.
+LLM-based evaluation can conveniently assess the correctness of model outputs (or other dimensions of metrics, requiring custom prompt settings). Below is an example using the preset `pattern` JSON contract to determine correctness.
 
 For a complete explanation of judge parameters, please refer to [documentation](../../get_started/parameters.md#judge-parameters).
 
+```{note}
+The judge replies with a single JSON object. A reply that cannot be read as one is **excluded** from
+the metric rather than scored 0, so `Num` may be lower than the sample count. A
+custom `prompt_template` should state the grading criteria only; the reply format is appended
+automatically.
+```
+
 ```python
 import os
 from evalscope import TaskConfig, run_task
-from evalscope.constants import JudgeStrategy
 
 task_cfg = TaskConfig(
     model='Qwen/Qwen2.5-0.5B-Instruct',
@@ -262,22 +268,18 @@ task_cfg = TaskConfig(
             ],
         }
     },
-    # judge related parameters
-    judge_model_args={
-        'model_id': 'qwen2.5-72b-instruct',
-        'api_url': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
-        'api_key': os.getenv('DASHSCOPE_API_KEY'),
-        'generation_config': {
-            'temperature': 0.0,
-            'max_tokens': 4096
+    judge={
+        'strategy': 'llm',
+        'models': {
+            'model_id': 'qwen2.5-72b-instruct',
+            'api_url': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
+            'api_key': os.getenv('DASHSCOPE_API_KEY'),
+            'generation_config': {'temperature': 0.0, 'max_tokens': 4096},
         },
-        # Determine if the model output is correct based on reference answers and model output
-        'score_type': 'pattern',
+        'contract': {'score_type': 'pattern'},
     },
     # eval concurrency number
     eval_batch_size=5,
-    # Use LLM for evaluation
-    judge_strategy=JudgeStrategy.LLM,
 )
 
 run_task(task_cfg=task_cfg)
@@ -298,13 +300,12 @@ run_task(task_cfg=task_cfg)
 
 If the dataset lacks reference answers, an LLM judge can be used to evaluate the model's output answers. Without configuring an LLM, no scoring results will be available.
 
-Below is an example configuring `judge_model_args` parameters, using the preset `numeric` mode to automatically assess model output scores from dimensions such as accuracy, relevance, and usefulness. Higher scores indicate better model output.
+Below is an example using the preset `numeric` JSON contract to automatically assess model output scores from dimensions such as accuracy, relevance, and usefulness. Higher scores indicate better model output.
 
 For a complete explanation of judge parameters, please refer to [documentation](../../get_started/parameters.md#judge-parameters).
 ```python
 import os
 from evalscope import TaskConfig, run_task
-from evalscope.constants import JudgeStrategy
 
 task_cfg = TaskConfig(
     model='Qwen/Qwen2.5-0.5B-Instruct',
@@ -319,22 +320,18 @@ task_cfg = TaskConfig(
             ],
         }
     },
-    # judge related parameters
-    judge_model_args={
-        'model_id': 'qwen2.5-72b-instruct',
-        'api_url': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
-        'api_key': os.getenv('DASHSCOPE_API_KEY'),
-        'generation_config': {
-            'temperature': 0.0,
-            'max_tokens': 4096
+    judge={
+        'strategy': 'llm',
+        'models': {
+            'model_id': 'qwen2.5-72b-instruct',
+            'api_url': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
+            'api_key': os.getenv('DASHSCOPE_API_KEY'),
+            'generation_config': {'temperature': 0.0, 'max_tokens': 4096},
         },
-        # Direct scoring
-        'score_type': 'numeric',
+        'contract': {'score_type': 'numeric'},
     },
     # eval concurrency number
     eval_batch_size=5,
-    # Use LLM for evaluation
-    judge_strategy=JudgeStrategy.LLM,
 )
 
 run_task(task_cfg=task_cfg)

docs/en/advanced_guides/custom_dataset/vlm.md

@@ -266,7 +266,7 @@ You can specify a judge model through the `judge_model` parameter to generate re
 
 ```python
 from evalscope.run import run_task
-from evalscope.constants import EvalType, JudgeStrategy
+from evalscope.constants import EvalType
 from os import environ as env
 
 task_cfg = TaskConfig(
@@ -282,17 +282,16 @@ task_cfg = TaskConfig(
         }
     },
     limit=5,
-    judge_model_args={
-        'model_id': 'qwen-plus',  # Does not need to be a multimodal model
-        'api_url': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
-        'api_key': env.get('DASHSCOPE_API_KEY'),
-        'generation_config': {
-            'temperature': 0.0,
-            'max_tokens': 4096
+    judge={
+        'strategy': 'llm',
+        'models': {
+            'model_id': 'qwen-plus',  # Does not need to be a multimodal model
+            'api_url': 'https://dashscope.aliyuncs.com/compatible-mode/v1',
+            'api_key': env.get('DASHSCOPE_API_KEY'),
+            'generation_config': {'temperature': 0.0, 'max_tokens': 4096},
         },
     },
     eval_batch_size=5,
-    judge_strategy=JudgeStrategy.LLM,
 )
 result = run_task(task_cfg=task_cfg)
 ```
@@ -307,9 +306,8 @@ evalscope eval \
   --datasets general_vqa \
   --dataset-args '{"general_vqa": {"local_path": "custom_eval/multimodal/vqa", "subset_list": ["example_openai"]}}' \
   --limit 5 \
-  --judge-model-args '{"model_id": "qwen-plus", "api_url": "https://dashscope.aliyuncs.com/compatible-mode/v1", "api_key": "$DASHSCOPE_API_KEY", "generation_config": {"temperature": 0.0, "max_tokens": 4096}}' \
-  --judge-worker-num 5 \
-  --judge-strategy llm
+  --eval-batch-size 5 \
+  --judge '{"strategy": "llm", "models": {"model_id": "qwen-plus", "api_url": "https://dashscope.aliyuncs.com/compatible-mode/v1", "api_key": "$DASHSCOPE_API_KEY", "generation_config": {"temperature": 0.0, "max_tokens": 4096}}}'
 ```
 
 Evaluation will output accuracy metrics:

docs/en/benchmarks/air_bench_chat.md

@@ -10,39 +10,22 @@ AIR-Bench Chat is the generative half of [AIR-Bench](https://arxiv.org/abs/2402.
 - **Task Type**: Open-ended audio question answering.
 - **Input**: An audio clip plus a free-form question.
 - **Output**: A textual answer evaluated against the reference response.
+- **Modalities**: Audio (human speech, natural sounds, music) + text.
 
-## Categories (8 tasks → 5 reported categories)
+## Key Features
 
-The 8 Chat tasks are aggregated by the official `cal_score.py` into five categories:
+- ~2k open-ended audio QA pairs across speech, sound, music and mixed-audio scenes; the generative half of AIR-Bench (ACL 2024).
+- 8 Chat tasks aggregated by the official `cal_score.py` into 5 reported categories: `speech` (`speech_QA`, `speech_dialogue_QA`), `sound` (`sound_QA`, `sound_generation_QA`), `music` (`music_QA`, `music_generation_analysis_QA`), `speech_and_sound` (`speech_and_sound_QA`), `speech_and_music` (`speech_and_music_QA`). The paper's Mixed-audio = mean(speech_and_sound, speech_and_music).
+- Position bias is removed by judging every sample twice with reference/prediction order swapped, then averaging (disable via `extra_params={'do_swap': False}` to halve judge cost).
+- Hosted on ModelScope ([`evalscope/AIR-Bench-Dataset`](https://modelscope.cn/datasets/evalscope/AIR-Bench-Dataset)) in an audiofolder + JSON layout; the full release is ~49 GB, so limit tasks via `extra_params={'tasks': [...]}` for partial runs.
 
-- `speech`: `speech_QA`, `speech_dialogue_QA`
-- `sound`: `sound_QA`, `sound_generation_QA`
-- `music`: `music_QA`, `music_generation_analysis_QA`
-- `speech_and_sound`: `speech_and_sound_QA`
-- `speech_and_music`: `speech_and_music_QA`
+## Evaluation Notes
 
-The paper's **Mixed-audio = mean(speech_and_sound, speech_and_music)**.
-
-## Dataset Access
-
-- The dataset is hosted on ModelScope: [`evalscope/AIR-Bench-Dataset`](https://modelscope.cn/datasets/evalscope/AIR-Bench-Dataset). It uses an *audiofolder + JSON metadata* layout. evalscope downloads it lazily via `modelscope.dataset_snapshot_download` on first run; the full release is ~49 GB, so it is recommended to limit which tasks are pulled via `extra_params`.
+- Metrics: `judge_score` is the model's mean judge score; `win_rate` records how often the model strictly beats the reference.
+- The judge LLM receives the question, the textual audio description (`meta_info`), the reference answer (`answer_gt`), and the model's response, and outputs two integer scores in `[1, 10]`. Use a judge that supports long contexts, since `meta_info` may exceed 4k tokens for dialogue tasks.
+- The official leaderboard uses `gpt-4-0125-preview`. If that exact snapshot is unavailable, use an available GPT-4-class judge; absolute scores can drift versus the published numbers because the judge model changed.
 - If the dataset is already on disk, pass `dataset_args={'air_bench_chat': {'local_path': '/path/to/AIR-Bench-Dataset'}}`; the local root should contain `Chat/`.
 
-## Evaluation Protocol
-
-- The judge LLM (default: GPT-4) receives the question, the textual audio description (`meta_info` from the dataset), the reference answer (`answer_gt`), and the model's response. It outputs a single line with two integer scores in `[1, 10]`.
-- To remove position bias, every sample is judged twice with the order of reference and prediction swapped, then averaged. This mirrors `cal_score.py` in the official repository — disable it via `extra_params={'do_swap': False}` to halve judge cost.
-- Reported metric `judge_score` is the model's mean judge score; `win_rate` records how often the model strictly beats the reference.
-
-```{warning}
-The official leaderboard uses `gpt-4-0125-preview` as the judge model. If that exact snapshot is unavailable, use an available GPT-4-class judge; absolute scores can drift versus the published numbers because the judge model changed.
-```
-
-## Implementation Notes
-
-- The judge model is selected via `--judge-model-args`; ensure the model id supports long contexts (`meta_info` may exceed 4k tokens for dialogue tasks).
-- Set `extra_params={'tasks': [...]}` to evaluate only specific Chat task names — useful for partial runs.
-
 
 ## Properties
 

docs/en/benchmarks/baby_vision.md

@@ -30,7 +30,7 @@ recognition, and visual tracking.
 - Primary metric: **Accuracy** via LLM-as-judge
 - Subsets organized by `type` field (4 categories)
 - LLM judge evaluates both choice and blank answer types uniformly
-- Requires `judge_model_args` configuration for LLM judge
+- Requires an LLM judge configured through `judge.models`
 
 
 ## Properties
@@ -48,30 +48,7 @@ recognition, and visual tracking.
 
 ## Data Statistics
 
-| Metric | Value |
-|--------|-------|
-| Total Samples | 388 |
-| Prompt Length (Mean) | 167.37 chars |
-| Prompt Length (Min/Max) | 33 / 450 chars |
-
-**Per-Subset Statistics:**
-
-| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |
-|--------|---------|-------------|------------|------------|
-| `Fine-grained Discrimination` | 163 | 152.09 | 33 | 450 |
-| `Spatial Perception` | 91 | 157.18 | 73 | 370 |
-| `Visual Pattern Recognition` | 51 | 178.92 | 94 | 319 |
-| `Visual Tracking` | 83 | 201.45 | 97 | 389 |
-
-**Image Statistics:**
-
-| Metric | Value |
-|--------|-------|
-| Total Images | 388 |
-| Images per Sample | min: 1, max: 1, mean: 1 |
-| Resolution Range | 174x144 - 2378x1448 |
-| Formats | jpeg, png, webp |
-
+*Statistics not available.*
 
 ## Sample Example
 
@@ -81,13 +58,13 @@ recognition, and visual tracking.
 {
   "input": [
     {
-      "id": "8b83904b",
+      "id": "1256c961",
       "content": [
         {
           "image": "[BASE64_IMAGE: jpeg, ~77.0KB]"
         },
         {
-          "text": "The image shows a total of 49 tiger patterns arranged in 7 rows and 7 columns. One of them is different from the others. Which row and column is it in? The answer format is (x,y). (For example, the answer for the 2nd row and 3rd column is (2,3))."
+          "text": "The image shows a total of 49 tiger patterns arranged in 7 rows and 7 columns. One of them is different from the others. Which row and column is it in? The answer format is (x,y). (For example, the answer for the 2nd row and 3rd column is (2,3)).\nThink about the question and give your final answer in \\boxed{Answer} format."
         }
       ]
     }
@@ -134,11 +111,6 @@ task_cfg = TaskConfig(
     api_url='OPENAI_API_COMPAT_URL',
     api_key='EMPTY_TOKEN',
     datasets=['baby_vision'],
-    dataset_args={
-        'baby_vision': {
-            # subset_list: ['Fine-grained Discrimination', 'Spatial Perception', 'Visual Pattern Recognition']  # optional, evaluate specific subsets
-        }
-    },
     limit=10,  # Remove this line for formal evaluation
 )