[Kh4L]

GDPVal currently prints transient judge failures such as HTTP 429 responses, but loses the request shape and whether a later retry recovered. This adds opt-in structured telemetry through judge_telemetry_output_dir, recording preflight exclusions, individual provider failures, retry recovery, invalid responses, and terminal matchup failures.

Each resources-server process writes its own JSONL file. Failure events include the task/reference/judge identity plus the actual transmitted file labels, types, counts, attachment sizes, request size, and transport limits. Compact completion events provide the denominator needed to calculate failure and recovery rates. Prompts, attachment contents, base64 payloads, credentials, and absolute paths are excluded, and telemetry write failures never affect judging.

The included summarizer reports failure classes, HTTP statuses, recovery and terminal-failure rates, preflight reasons, request-size buckets, judges, tasks, and file extensions:

python -m resources_servers.gdpval.judge_telemetry /path/to/judge_telemetry

Judging behavior, retries, routing, scoring, and ELO fitting are unchanged when telemetry is disabled or enabled.

Validation:

• 159 affected GDPVal tests passed
• scoped pre-commit hooks passed

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.