diff --git a/CHANGELOG.md b/CHANGELOG.md index 01ddd932e..9f9c3ddcb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,23 @@ All notable changes to SkillEvaluator are documented in this file. ### Added +- Transparent HTTP 429 (rate-limiting), transient 5xx, and timeout recovery for + LLM judges in both the Harbor container verifier (`eval.py`) and host runtime + (`LLMClient`). Features zero-dependency full jitter exponential backoff, + RFC-7231 `Retry-After` header parsing, finite-value environment overrides + (`SKILL_EVAL_LLM_MAX_RETRIES`, `SKILL_EVAL_LLM_RETRY_BASE_DELAY`, and + `SKILL_EVAL_LLM_RETRY_MAX_DELAY`), and + automatic container forwarding via Harbor `task.toml`, and a per-judge + verifier time budget that leaves room for failure artifacts, without altering + benchmark metrics or scoring formulas. +- Provider-aware structured JSON schema enforcement (`response_format` for + OpenAI-compatible / Gemini Vertex / NVIDIA NIM endpoints and `output_config` + for Anthropic `/v1/messages`) across the custom `judge_accuracy`, + `judge_goal_accuracy`, and `judge_behavior_check` paths, with automatic + schema-specific `HTTP 400`/`422` downgrade and per-target memoization + (`_SCHEMA_UNSUPPORTED_TARGETS`), boolean prompt alignment, and a guard for + missing `message` fields on reasoning token exhaustion. The canonical OpenAI + RAGAS goal scorer retains its separate scoring path. - Configurable evidence bundle budgets (`SKILL_EVAL_ACCURACY_BUDGET`, `SKILL_EVAL_GOAL_ACCURACY_BUDGET`, `SKILL_EVAL_BEHAVIOR_CHECK_BUDGET`) and final response limit (`SKILL_EVAL_BEHAVIOR_FINAL_RESPONSE_LIMIT`). ## 0.4.0 - 2026-09-30 diff --git a/docs/environment-variables.mdx b/docs/environment-variables.mdx index e3d49ea91..0ef64bf74 100644 --- a/docs/environment-variables.mdx +++ b/docs/environment-variables.mdx @@ -29,6 +29,10 @@ These variables select and configure the provider used for LLM-backed checks and | `SKILL_EVAL_LLM_MODEL` | OpenAI: `gpt-5.6-sol`; Anthropic: `claude-opus-5`; NVIDIA Build: `nvidia/nemotron-3-super-120b-a12b`; OpenAI-compatible: `nvidia/nvidia/nemotron-3-super-120b-long-ctx`; Bedrock: `us.anthropic.claude-opus-5` | Chat model override. For gateways, use the exact catalog ID when it differs from the default. Setting it to an empty string is a configuration error. For a lower-cost OpenAI option, use `gpt-5.4-mini`. | | `SKILL_EVAL_LLM_BASE_URL` | provider default endpoint | Endpoint override; takes precedence over `OPENAI_BASE_URL` and `ANTHROPIC_BASE_URL`. Required for `openai-compatible`. Ignored by `nv_build`, whose endpoint is fixed to `https://integrate.api.nvidia.com/v1` — point a custom endpoint at the `openai-compatible` provider instead. | | `SKILL_EVAL_LLM_API_KEY` | — | API key for the `openai-compatible` provider (local servers still require it to be set). | +| `SKILL_EVAL_LLM_MAX_RETRIES` | `3` | Extra attempts for transient LLM judge failures after the initial request. `0` disables retries; invalid or negative values use the default. | +| `SKILL_EVAL_LLM_RETRY_BASE_DELAY` | `1` second | Base for exponential jitter between direct LLM judge attempts. Values must be finite and nonnegative. | +| `SKILL_EVAL_LLM_RETRY_MAX_DELAY` | `30` seconds | Maximum sleep between direct LLM judge attempts. A `Retry-After` value above this limit fails fast; values must be finite and nonnegative. | +| `SKILL_EVAL_LLM_JUDGE_BUDGET_SEC` | `180` seconds | Wall-time budget per required LLM judge in the Harbor verifier. Values must be finite and positive; invalid or nonpositive values use the default. | | `SKILL_EVAL_MODEL_CATALOG_ALLOW_HTTP_HOSTS` | unset | Comma-separated hosts whose model catalog may be read over plain HTTP. Catalog reads otherwise require HTTPS unless the host is loopback, because the request carries a bearer token. Use this only when the transport is already encrypted and authenticated below HTTP — a WireGuard or comparable tunnel peer, for example — where the tool cannot see that the link is protected. Each entry matches one whole host as written, with no resolution, no suffix matching, and no wildcards. A plain-HTTP request to an accepted host also bypasses any inherited HTTP proxy, so the bearer token is never offered to an intermediary. The transport rechecks authorization before dispatch and rejects hosts that are no longer allowed. HTTPS routing is unchanged. | | `NVIDIA_API_KEY` | — | Credential for the `nv_build` provider (NVIDIA Build). | | `OPENAI_API_KEY` | — | Credential for the `openai` provider. | @@ -38,6 +42,12 @@ These variables select and configure the provider used for LLM-backed checks and | `AWS_REGION` | `us-west-2` | Region for the `bedrock` provider. | | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_PROFILE`, ... | — | Standard AWS credential chain, used as-is by the `bedrock` provider. | +The retry settings apply to the shared host LLM client and direct public-provider +calls in the Harbor verifier. The canonical OpenAI RAGAS goal scorer uses its +own adapter and scoring path. Harbor's managed verifier limits each required +judge to `SKILL_EVAL_LLM_JUDGE_BUDGET_SEC` (180 seconds by default) so retries +leave time to write a failed-run report. + ### Anthropic endpoint roots When `anthropic` is the selected evaluator provider, endpoint precedence is `SKILL_EVAL_LLM_BASE_URL`, then `ANTHROPIC_BASE_URL`, then the Anthropic default. A root such as `https://gateway.example.com` and a legacy terminal `/v1` root such as `https://gateway.example.com/team/v1` both produce request paths with exactly one `/v1/messages` suffix for the Anthropic SDK, the Tier 3 verifier, and a matching Claude Code agent. An independently credentialed Claude Code route applies the same normalization and validation to `ANTHROPIC_BASE_URL` when the evaluator uses another provider. Routing through path prefixes, ports, IP addresses, internationalized domain names, and ordinary safe percent escapes is preserved. diff --git a/src/skillevaluator/inference/client.py b/src/skillevaluator/inference/client.py index 74faca9a8..e90ffd8b2 100644 --- a/src/skillevaluator/inference/client.py +++ b/src/skillevaluator/inference/client.py @@ -18,13 +18,17 @@ class supports two usage patterns: from __future__ import annotations +import io import json import os +import re +import urllib.error from dataclasses import replace -from typing import Any +from typing import Any, NamedTuple from urllib.parse import urlsplit from skillevaluator.constants import LLM_VERIFY_MODEL, LLM_VERIFY_TEMPERATURE +from skillevaluator.inference.retry import RetryConfig, resolve_retry_config, retry_call_with_backoff from skillevaluator.inference.types import EmptyLLMResponseError, LLMClientError from skillevaluator.logging_config import get_logger from skillevaluator.provider_config import ( @@ -104,6 +108,171 @@ def _temperature_kwargs(model: str, temperature: float | None) -> dict[str, floa return {"temperature": temperature} +class SchemaTargetKey(NamedTuple): + """Identify a provider, base URL, and model target for schema capability memoization.""" + + provider: str + base_url: str + model: str + + +_SCHEMA_UNSUPPORTED_TARGETS: set[SchemaTargetKey | tuple[str, str, str]] = set() + + +def _build_openai_response_format(schema: dict[str, Any], schema_name: str = "judge_response") -> dict[str, Any]: + """Build OpenAI-compatible JSON schema response_format payload.""" + return { + "type": "json_schema", + "json_schema": { + "name": schema_name, + "strict": True, + "schema": schema, + }, + } + + +def _build_anthropic_output_config(schema: dict[str, Any]) -> dict[str, Any]: + """Build Anthropic Messages API output_config payload.""" + return { + "format": { + "type": "json_schema", + "schema": schema, + } + } + + +_UNSUPPORTED_REASON_INDICATORS: tuple[str, ...] = ( + "unsupported", + "not supported", + "extra input", + "extra inputs", + "unknown parameter", + "unknown field", + "unknown argument", + "unrecognized request argument", + "unrecognized parameter", + "unexpected keyword argument", + "unexpected argument", + "invalid parameter", + "invalid argument", + "not permitted", + "not allowed", + "disallowed", +) + +_SCHEMA_OPTION_PATTERN = r"(?:response_format|response format|output_config|json_schema|structured[_ ]outputs?)" +_SCHEMA_REJECTION_REASON = ( + r"(?:unsupported|not supported|not permitted|not allowed|disallowed|" + r"unknown (?:parameter|field|argument)|unrecognized (?:request argument|parameter)|" + r"unexpected (?:keyword )?argument|extra inputs?(?: are not permitted)?)" +) +_SCHEMA_REJECTION_AFTER_OPTION = re.compile( + rf"\b{_SCHEMA_OPTION_PATTERN}\b(?:\.[a-z0-9_]+)*" + rf"(?:\s+of\s+type\s+['\"]?[a-z0-9_]+['\"]?)?" + rf"\s*(?:(?:is|are|was|were)\s+(?:an?\s+)?|:\s*)?" + rf"{_SCHEMA_REJECTION_REASON}\b", + re.IGNORECASE, +) +_SCHEMA_REJECTION_BEFORE_OPTION = re.compile( + rf"\b(?:unsupported|not supported|extra inputs?(?: are not permitted)?|unknown (?:parameter|field|argument)|" + rf"unrecognized (?:request argument|parameter)|unexpected (?:keyword argument|argument)|" + rf"invalid (?:parameter|argument)|not permitted|not allowed|disallowed)\b" + rf"(?:\s+supplied)?[\s:'\"\[\]{{}}(),-]{{0,32}}\b{_SCHEMA_OPTION_PATTERN}\b", + re.IGNORECASE, +) + + +def _message_rejects_schema_option(text: str, param: str | None = None) -> bool: + """Match a rejection of the schema option itself, not unrelated error text.""" + if param: + if not re.search(rf"\b{_SCHEMA_OPTION_PATTERN}\b", param, re.IGNORECASE): + return False + return any(indicator in text.lower() for indicator in _UNSUPPORTED_REASON_INDICATORS) + return bool(_SCHEMA_REJECTION_AFTER_OPTION.search(text) or _SCHEMA_REJECTION_BEFORE_OPTION.search(text)) + + +def _is_schema_unsupported_error(exc: Exception) -> bool: + """Determine whether an exception indicates structured output schema is unsupported.""" + status_code = getattr(exc, "status_code", None) + if status_code is None: + response = getattr(exc, "response", None) + status_code = getattr(response, "status_code", None) + if status_code is None and isinstance(exc, urllib.error.HTTPError): + status_code = exc.code + + is_type_error = isinstance(exc, TypeError) + if status_code not in {400, 422} and not is_type_error: + return False + + parts: list[str] = [str(exc), getattr(exc, "message", "")] + error_param: str | None = None + body = getattr(exc, "body", None) + if isinstance(body, dict): + parts.append(str(body)) + error_dict = body.get("error") + if isinstance(error_dict, dict): + parts.append(str(error_dict.get("message", ""))) + param = error_dict.get("param") + if isinstance(param, str): + error_param = param + elif isinstance(body, str): + parts.append(body) + + response = getattr(exc, "response", None) + if response is not None: + text = getattr(response, "text", None) + if isinstance(text, str): + parts.append(text) + + if isinstance(exc, urllib.error.HTTPError): + try: + body_bytes = exc.read() + exc.fp = io.BytesIO(body_bytes) + parts.append(body_bytes.decode("utf-8", "replace")) + except Exception: + pass + + full_text = " ".join(part for part in parts if part) + return _message_rejects_schema_option(full_text, error_param) + + +def _call_with_schema_fallback( + call_fn: Any, + call_kwargs: dict[str, Any], + *, + schema_key: str, + target_key: SchemaTargetKey, + use_schema: bool, +) -> Any: + """Invoke call_fn and downgrade to prompt-only on confirmed HTTP 400/422 schema errors.""" + try: + return call_fn(**call_kwargs) + except Exception as exc: + if use_schema and _is_schema_unsupported_error(exc): + logger.warning( + "Structured output schema unsupported by provider=%s model=%s; " + "downgrading to prompt-only JSON and memoizing target.", + target_key.provider, + target_key.model, + ) + fallback_kwargs = dict(call_kwargs) + fallback_kwargs.pop(schema_key, None) + result = call_fn(**fallback_kwargs) + _SCHEMA_UNSUPPORTED_TARGETS.add(target_key) + return result + raise + + +def _extract_choice_content(response: Any) -> str: + choices = getattr(response, "choices", None) or [] + first_choice = choices[0] if choices else None + message = getattr(first_choice, "message", None) if first_choice is not None else None + content = getattr(message, "content", None) if message is not None else "" + if not content: + return "" + return content.strip() + + class LLMClient: """Public-provider client for chat completions. @@ -132,12 +301,23 @@ def __init__( *, max_tokens: int | None = None, temperature: float | None = None, + max_retries: int | None = None, + retry_base_delay: float | None = None, + retry_max_delay: float | None = None, + http_client: Any = None, ) -> None: self._model = model self._base_url = base_url self._api_key = api_key self._max_tokens = max_tokens if max_tokens is not None else self.default_max_tokens self._temperature = temperature if temperature is not None else self.default_temperature + retry_cfg = resolve_retry_config() + self._retry_config = RetryConfig( + max_retries=max_retries if max_retries is not None else retry_cfg.max_retries, + base_delay=retry_base_delay if retry_base_delay is not None else retry_cfg.base_delay, + max_delay=retry_max_delay if retry_max_delay is not None else retry_cfg.max_delay, + ) + self._http_client = http_client self._client: Any = None self._provider_config: ProviderConfig | None = None @@ -159,6 +339,21 @@ def api_key(self) -> str | None: def temperature(self) -> float | None: return self._temperature + @property + def max_retries(self) -> int: + """Return the maximum number of retry attempts for transient errors.""" + return self._retry_config.max_retries + + @property + def retry_base_delay(self) -> float: + """Return the initial base backoff delay in seconds.""" + return self._retry_config.base_delay + + @property + def retry_max_delay(self) -> float: + """Return the maximum delay ceiling in seconds for a retry backoff.""" + return self._retry_config.max_delay + # -- client management ------------------------------------------------ def _resolved_config(self) -> ProviderConfig: @@ -208,9 +403,11 @@ def _get_client(self) -> Any: raise LLMClientError( "The 'anthropic' package is required for Anthropic LLM operations. Install with: pip install 'skillevaluator[llm]'" ) from exc - client_kwargs: dict[str, Any] = {"api_key": config.api_key} + client_kwargs: dict[str, Any] = {"api_key": config.api_key, "max_retries": 0} if config.base_url: client_kwargs["base_url"] = config.base_url + if self._http_client is not None: + client_kwargs["http_client"] = self._http_client self._client = Anthropic(**client_kwargs) return self._client @@ -221,7 +418,14 @@ def _get_client(self) -> Any: "The 'openai' package is required for LLM operations. Install it with: pip install openai" ) from exc - client = OpenAI(api_key=config.api_key, base_url=config.base_url) + client_kwargs: dict[str, Any] = { + "api_key": config.api_key, + "base_url": config.base_url, + "max_retries": 0, + } + if self._http_client is not None: + client_kwargs["http_client"] = self._http_client + client = OpenAI(**client_kwargs) if ( config.base_url is not None and not _is_canonical_openai_base_url(config.base_url) @@ -236,63 +440,94 @@ def _get_client(self) -> Any: # -- direct-use methods ----------------------------------------------- - def completions(self, system_prompt: str, user_prompt: str) -> str: + def completions( + self, + system_prompt: str, + user_prompt: str, + *, + response_schema: dict[str, Any] | None = None, + schema_name: str = "judge_response", + ) -> str: """Send a chat completion request and return the response text. Raises :class:`LLMClientError` when the response is empty. """ config = self._resolved_config() client = self._get_client() - if config.provider == "anthropic": + target_key = SchemaTargetKey(config.provider, config.base_url or "", config.model) + + def _invoke_provider() -> str: + use_schema = response_schema is not None and target_key not in _SCHEMA_UNSUPPORTED_TARGETS + if config.provider == "anthropic": + call_kwargs: dict[str, Any] = { + "model": config.model, + "max_tokens": self._max_tokens or 4096, + "system": system_prompt, + "messages": [{"role": "user", "content": user_prompt}], + **_temperature_kwargs(config.model, self._temperature), + } + if use_schema and response_schema is not None: + call_kwargs["output_config"] = _build_anthropic_output_config(response_schema) + response = _call_with_schema_fallback( + client.messages.create, + call_kwargs, + schema_key="output_config", + target_key=target_key, + use_schema=use_schema, + ) + content = "".join( + str(block.text) for block in response.content if getattr(block, "type", None) == "text" + ) + if not content: + raise EmptyLLMResponseError("LLM returned empty response content") + return content.strip() + if config.provider == "bedrock": + try: + from litellm import completion + except ImportError as exc: + raise LLMClientError( + "The 'litellm' package is required for Bedrock LLM operations. Install with: pip install 'skillevaluator[llm]'" + ) from exc + response = completion( + model=config.litellm_model, + messages=[ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_prompt}, + ], + aws_region_name=config.region, + **_temperature_kwargs(config.model, self._temperature), + **({"max_tokens": self._max_tokens} if self._max_tokens is not None else {}), + ) + content = _extract_choice_content(response) + if not content: + raise EmptyLLMResponseError("LLM returned empty response content") + return content call_kwargs: dict[str, Any] = { "model": config.model, - "max_tokens": self._max_tokens or 4096, - "system": system_prompt, - "messages": [{"role": "user", "content": user_prompt}], - **_temperature_kwargs(config.model, self._temperature), - } - response = client.messages.create(**call_kwargs) - content = "".join(str(block.text) for block in response.content if getattr(block, "type", None) == "text") - if not content: - raise EmptyLLMResponseError("LLM returned empty response content") - return content.strip() - if config.provider == "bedrock": - try: - from litellm import completion - except ImportError as exc: - raise LLMClientError( - "The 'litellm' package is required for Bedrock LLM operations. Install with: pip install 'skillevaluator[llm]'" - ) from exc - response = completion( - model=config.litellm_model, - messages=[ + "stream": False, + "messages": [ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}, ], - aws_region_name=config.region, **_temperature_kwargs(config.model, self._temperature), - **({"max_tokens": self._max_tokens} if self._max_tokens is not None else {}), + **_token_limit_kwargs(config, self._max_tokens), + } + if use_schema and response_schema is not None: + call_kwargs["response_format"] = _build_openai_response_format(response_schema, schema_name) + + response = _call_with_schema_fallback( + client.chat.completions.create, + call_kwargs, + schema_key="response_format", + target_key=target_key, + use_schema=use_schema, ) - content = response.choices[0].message.content + content = _extract_choice_content(response) if not content: raise EmptyLLMResponseError("LLM returned empty response content") - return str(content).strip() - call_kwargs: dict[str, Any] = { - "model": config.model, - "stream": False, - "messages": [ - {"role": "system", "content": system_prompt}, - {"role": "user", "content": user_prompt}, - ], - **_temperature_kwargs(config.model, self._temperature), - **_token_limit_kwargs(config, self._max_tokens), - } + return content - response = client.chat.completions.create(**call_kwargs) - content = response.choices[0].message.content - if not content: - raise EmptyLLMResponseError("LLM returned empty response content") - return content.strip() + return retry_call_with_backoff(_invoke_provider, config=self._retry_config) def extract_json_from_response(self, system_prompt: str, user_prompt: str) -> dict: """Send a completion and parse JSON from the response.""" diff --git a/src/skillevaluator/inference/retry.py b/src/skillevaluator/inference/retry.py new file mode 100644 index 000000000..473dc8c6f --- /dev/null +++ b/src/skillevaluator/inference/retry.py @@ -0,0 +1,195 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Provide zero-dependency exponential backoff with full jitter and header-aware retry logic.""" + +from __future__ import annotations + +import logging +import math +import os +import random +import time +import urllib.error +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from datetime import UTC, datetime +from email.utils import parsedate_to_datetime + +logger = logging.getLogger(__name__) + +DEFAULT_MAX_RETRIES: int = 3 +DEFAULT_BASE_DELAY: float = 1.0 +DEFAULT_MAX_DELAY: float = 30.0 +_RETRIABLE_HTTP_STATUS_CODES: frozenset[int] = frozenset({429, 500, 502, 503, 504}) +_MAX_RETRIES_ENV: str = "SKILL_EVAL_LLM_MAX_RETRIES" +_BASE_DELAY_ENV: str = "SKILL_EVAL_LLM_RETRY_BASE_DELAY" +_MAX_DELAY_ENV: str = "SKILL_EVAL_LLM_RETRY_MAX_DELAY" + + +@dataclass(frozen=True) +class RetryConfig: + """Hold configuration parameters for retry and exponential backoff.""" + + max_retries: int = DEFAULT_MAX_RETRIES + base_delay: float = DEFAULT_BASE_DELAY + max_delay: float = DEFAULT_MAX_DELAY + + +def is_retriable_status_code(status_code: int | None) -> bool: + """Return True if the given HTTP status code represents a transient, retriable condition.""" + return status_code in _RETRIABLE_HTTP_STATUS_CODES + + +def parse_retry_after(header_value: str | None, fallback_delay: float) -> float: + """Parse a Retry-After header as seconds or HTTP date, falling back to default.""" + if not header_value: + return fallback_delay + clean_val = str(header_value).strip() + try: + return max(0.0, float(clean_val)) + except ValueError: + pass + try: + target = parsedate_to_datetime(clean_val) + if target.tzinfo is None: + target = target.replace(tzinfo=UTC) + now = datetime.now(UTC) + return max(0.0, (target - now).total_seconds()) + except Exception: + return fallback_delay + + +def calculate_full_jitter_delay( + attempt: int, + base_delay: float = DEFAULT_BASE_DELAY, + max_delay: float = DEFAULT_MAX_DELAY, + factor: float = 2.0, +) -> float: + """Calculate delay using exponential backoff with full jitter (AWS standard).""" + backoff = min(max_delay, base_delay * (factor**attempt)) + return random.uniform(0.0, backoff) + + +def _int_env(environ: Mapping[str, str], name: str, fallback: int) -> int: + """Read a non-negative integer from the environment variable or return default.""" + raw = str(environ.get(name, "")).strip() + if raw: + try: + val = int(raw) + return val if val >= 0 else fallback + except ValueError: + return fallback + return fallback + + +def _float_env(environ: Mapping[str, str], name: str, fallback: float) -> float: + """Read a non-negative float from the environment variable or return default.""" + raw = str(environ.get(name, "")).strip() + if raw: + try: + val = float(raw) + return val if math.isfinite(val) and val >= 0.0 else fallback + except ValueError: + return fallback + return fallback + + +def resolve_retry_config(environ: Mapping[str, str] | None = None) -> RetryConfig: + """Resolve retry and backoff limits from environment variables with safe defaults.""" + env = os.environ if environ is None else environ + max_retries = _int_env(env, _MAX_RETRIES_ENV, fallback=DEFAULT_MAX_RETRIES) + base_delay = _float_env(env, _BASE_DELAY_ENV, fallback=DEFAULT_BASE_DELAY) + raw_max_delay = _float_env(env, _MAX_DELAY_ENV, fallback=DEFAULT_MAX_DELAY) + max_delay = max(base_delay, raw_max_delay) + return RetryConfig(max_retries=max_retries, base_delay=base_delay, max_delay=max_delay) + + +def extract_http_status(exc: BaseException) -> int | None: + """Extract numeric HTTP status code from an exception if available.""" + if isinstance(exc, urllib.error.HTTPError): + return getattr(exc, "code", None) + if hasattr(exc, "status_code") and isinstance(exc.status_code, int): + return exc.status_code + if hasattr(exc, "status") and isinstance(exc.status, int): + return exc.status + return None + + +def extract_retry_after(exc: BaseException) -> str | None: + """Extract Retry-After header string from an exception if present.""" + if isinstance(exc, urllib.error.HTTPError) and exc.headers: + return exc.headers.get("retry-after") or exc.headers.get("Retry-After") + response = getattr(exc, "response", None) + if response is not None and hasattr(response, "headers") and response.headers: + return response.headers.get("retry-after") or response.headers.get("Retry-After") + return None + + +def is_retriable_exception(exc: BaseException) -> bool: + """Return True if an exception represents a transient failure eligible for retry.""" + status = extract_http_status(exc) + if status is not None: + return is_retriable_status_code(status) + + if isinstance(exc, (TimeoutError, urllib.error.URLError, ConnectionError, OSError)): + return True + + type_name = type(exc).__name__ + return any(keyword in type_name for keyword in ("RateLimit", "APIConnection", "Timeout", "InternalServer")) + + +def retry_call_with_backoff[T]( + operation: Callable[[], T], + *, + config: RetryConfig | None = None, + max_retries: int | None = None, + base_delay: float | None = None, + max_delay: float | None = None, + on_retry: Callable[[BaseException, int, float], None] | None = None, +) -> T: + """Execute operation, retrying on transient HTTP or network faults using full jitter.""" + cfg = config if config is not None else resolve_retry_config() + retries_limit = cfg.max_retries if max_retries is None else max(0, max_retries) + ceiling_delay = cfg.max_delay if max_delay is None else max(0.0, max_delay) + initial_delay = min(ceiling_delay, cfg.base_delay if base_delay is None else max(0.0, base_delay)) + + attempt = 0 + + while True: + try: + return operation() + except Exception as exc: + if attempt >= retries_limit or not is_retriable_exception(exc): + raise + + retry_after_str = extract_retry_after(exc) + if retry_after_str is not None: + parsed_delay = parse_retry_after(retry_after_str, fallback_delay=initial_delay) + if parsed_delay > ceiling_delay: + raise + delay = parsed_delay + random.uniform(0.1, 0.5) + else: + delay = calculate_full_jitter_delay( + attempt=attempt, + base_delay=initial_delay, + max_delay=ceiling_delay, + ) + + sleep_duration = min(delay, ceiling_delay) + + if on_retry is not None: + on_retry(exc, attempt + 1, sleep_duration) + else: + status = extract_http_status(exc) + status_label = f"HTTP {status}" if status is not None else type(exc).__name__ + logger.warning( + "LLM call encountered transient error (%s). Retrying in %.2fs (attempt %d/%d)...", + status_label, + sleep_duration, + attempt + 1, + retries_limit, + ) + + time.sleep(sleep_duration) + attempt += 1 diff --git a/src/skillevaluator/tier3/eval_core/llm_judge.py b/src/skillevaluator/tier3/eval_core/llm_judge.py index 0310e1130..f3a4c3323 100644 --- a/src/skillevaluator/tier3/eval_core/llm_judge.py +++ b/src/skillevaluator/tier3/eval_core/llm_judge.py @@ -209,6 +209,8 @@ def _chat_completion_payload( temperature: float, provider: str | None = None, request_url: str | None = None, + response_schema: dict[str, Any] | None = None, + schema_name: str = "judge_response", ) -> dict[str, Any]: resolved_provider = _provider() if provider is None else provider resolved_request_url = _resolve_url(resolved_provider) if request_url is None else request_url @@ -226,6 +228,10 @@ def _chat_completion_payload( } if temperature is not None and _supports_custom_temperature(model): payload["temperature"] = temperature + if response_schema is not None: + from skillevaluator.inference.client import _build_openai_response_format + + payload["response_format"] = _build_openai_response_format(response_schema, schema_name) return payload @@ -238,6 +244,8 @@ def call_public_llm( temperature: float = 0.0, timeout: int = 60, allow_model_fallback: bool = True, + response_schema: dict[str, Any] | None = None, + schema_name: str = "judge_response", ) -> tuple[str | None, str | None]: """Call the configured public provider through the shared client. @@ -255,7 +263,15 @@ def call_public_llm( max_tokens=max_tokens, temperature=temperature, ) - return client.completions("You are a precise evaluation judge.", prompt), None + return ( + client.completions( + "You are a precise evaluation judge.", + prompt, + response_schema=response_schema, + schema_name=schema_name, + ), + None, + ) except EmptyLLMResponseError: return "", None except Exception as exc: @@ -608,6 +624,7 @@ def _call_validated_json_judge( extract: Any, **call_kwargs: Any, ) -> tuple[Any, str | None, dict[str, Any]]: + """Invoke a JSON judge with one format-correction retry when payload validation fails.""" call_kwargs.setdefault("max_tokens", STRUCTURED_JUDGE_MAX_TOKENS) def invoke(call_prompt: str) -> tuple[Any, str | None, dict[str, Any], str | None]: @@ -635,8 +652,72 @@ def invoke(call_prompt: str) -> tuple[Any, str | None, dict[str, Any], str | Non # Accuracy judge (5-criterion) # --------------------------------------------------------------------------- +ACCURACY_JSON_SCHEMA: dict[str, Any] = { + "type": "object", + "properties": { + "criteria": { + "type": "object", + "properties": { + "SKILL_IDENTIFIED": {"type": "boolean"}, + "ACTION_CORRECT": {"type": "boolean"}, + "FACTUALLY_ACCURATE": {"type": "boolean"}, + "TASK_ADDRESSED": {"type": "boolean"}, + "ACTIONABLE": {"type": "boolean"}, + }, + "required": [ + "SKILL_IDENTIFIED", + "ACTION_CORRECT", + "FACTUALLY_ACCURATE", + "TASK_ADDRESSED", + "ACTIONABLE", + ], + "additionalProperties": False, + }, + "score": {"type": "number"}, + "reason": {"type": "string"}, + }, + "required": ["criteria", "score", "reason"], + "additionalProperties": False, +} + +GOAL_ACCURACY_JSON_SCHEMA: dict[str, Any] = { + "type": "object", + "properties": { + "user_goal": {"type": "string"}, + "end_state": {"type": "string"}, + "achieved": {"type": "boolean"}, + "score": {"type": "number"}, + "reason": {"type": "string"}, + }, + "required": ["user_goal", "end_state", "achieved", "score", "reason"], + "additionalProperties": False, +} + +BEHAVIOR_CHECK_JSON_SCHEMA: dict[str, Any] = { + "type": "object", + "properties": { + "results": { + "type": "array", + "items": { + "type": "object", + "properties": { + "step": {"type": "integer"}, + "passed": {"type": "boolean"}, + "reason": {"type": "string"}, + }, + "required": ["step", "passed", "reason"], + "additionalProperties": False, + }, + }, + "score": {"type": "number"}, + "summary": {"type": "string"}, + }, + "required": ["results", "score", "summary"], + "additionalProperties": False, +} + ACCURACY_PROMPT = """You are an expert evaluator for AI agent responses. Evaluate by checking \ -each criterion below against the expected answer. For each, answer YES or NO. +each criterion below against the expected answer. For each criterion, determine true (satisfied) or false (not satisfied). 1. SKILL_IDENTIFIED: Does the response reference or use the correct skill for the task? 2. ACTION_CORRECT: Does the response describe or execute the correct actions/scripts? @@ -644,8 +725,7 @@ def invoke(call_prompt: str) -> tuple[Any, str | None, dict[str, Any], str | Non 4. TASK_ADDRESSED: Does the response directly address the user's request? 5. ACTIONABLE: Does the response provide actionable information (not just acknowledgment)? -For each criterion write: YES or NO with a brief reason. -Then compute score = count(YES) / 5. +Compute score = count(true) / 5. Be lenient on exact wording but strict on factual correctness. Respond with ONLY a JSON object: @@ -674,6 +754,7 @@ def invoke(call_prompt: str) -> tuple[Any, str | None, dict[str, Any], str | Non def _valid_accuracy_criteria(value: Any) -> bool: + """Return True when value is a complete 5-criterion boolean mapping.""" return ( isinstance(value, dict) and value.keys() == _ACCURACY_CRITERIA_KEYS @@ -682,6 +763,7 @@ def _valid_accuracy_criteria(value: Any) -> bool: def _accuracy_payload_error(parsed: Any) -> str | None: + """Validate a parsed accuracy judge payload and return an error message if malformed.""" if not isinstance(parsed, dict): return "Judge response was not a valid JSON object" if "reason" in parsed and not isinstance(parsed["reason"], str): @@ -710,6 +792,8 @@ def judge_accuracy( ground_truth=ground_truth, agent_text=agent_text, ) + kwargs.setdefault("response_schema", ACCURACY_JSON_SCHEMA) + kwargs.setdefault("schema_name", "accuracy_judgment") parsed, error, _provenance = _call_validated_json_judge( prompt, @@ -769,6 +853,7 @@ def judge_accuracy( def _goal_payload_error(parsed: Any) -> str | None: + """Validate a parsed goal-accuracy judge payload and return an error message if malformed.""" if not isinstance(parsed, dict): return "Judge response was not a valid JSON object" for field in ("reason", "user_goal", "end_state"): @@ -798,6 +883,8 @@ def judge_goal_accuracy( tool_summary=tool_summary, agent_text=agent_text, ) + kwargs.setdefault("response_schema", GOAL_ACCURACY_JSON_SCHEMA) + kwargs.setdefault("schema_name", "goal_accuracy_judgment") parsed, error, _provenance = _call_validated_json_judge( prompt, @@ -840,7 +927,7 @@ def judge_goal_accuracy( EXPECTED BEHAVIORS: {behaviors} -For each behavior, respond YES (observed) or NO (not observed) with a brief reason. +For each behavior, set "passed" to true (observed) or false (not observed) with a brief reason. Respond with ONLY a JSON object: {{"results": [{{"step": 1, "passed": true/false, "reason": "..."}}, ...], \ @@ -959,12 +1046,15 @@ def judge_behavior_check( behaviors=behaviors_text, ) kwargs.setdefault("max_tokens", BEHAVIOR_JUDGE_MAX_TOKENS) + kwargs.setdefault("response_schema", BEHAVIOR_CHECK_JSON_SCHEMA) + kwargs.setdefault("schema_name", "behavior_check_judgment") content, error = call_public_llm(prompt, **kwargs) if error: return _judge_error(f"LLM judge error: {error}", results=[]) - def _parse_judge_object(text: str) -> dict[str, Any] | list[Any] | None: + def _parse_judge_object(text: str | None) -> dict[str, Any] | list[Any] | None: + """Parse a JSON object or list from judge response text.""" return _extract_json(text) if text else None parsed = _parse_judge_object(content) diff --git a/src/skillevaluator/tier3/harbor/adapter.py b/src/skillevaluator/tier3/harbor/adapter.py index 852433815..2438c8388 100644 --- a/src/skillevaluator/tier3/harbor/adapter.py +++ b/src/skillevaluator/tier3/harbor/adapter.py @@ -211,6 +211,14 @@ "SKILL_EVAL_GOAL_ACCURACY_BUDGET", } ) +_VERIFIER_RETRY_ENV_VARS = frozenset( + { + "SKILL_EVAL_LLM_JUDGE_BUDGET_SEC", + "SKILL_EVAL_LLM_MAX_RETRIES", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY", + } +) _VERIFIER_PROVIDER_ENV_VARS = frozenset( { "ANTHROPIC_API_KEY", @@ -236,6 +244,7 @@ "OPENAI_API_KEY", "OPENAI_BASE_URL", *_VERIFIER_BUDGET_ENV_VARS, + *_VERIFIER_RETRY_ENV_VARS, "SKILL_EVAL_LLM_API_KEY", "SKILL_EVAL_LLM_BASE_URL", "SKILL_EVAL_LLM_MODEL", @@ -2344,10 +2353,11 @@ def _resolve_docker_source_variables(source: str, defaults: dict[str, str | None def _replace(match: re.Match[str]) -> str: nonlocal changed name = match.group("braced") or match.group("plain") - if name not in defaults or defaults[name] is None: + value = defaults.get(name) if name is not None else None + if value is None: return match.group(0) changed = True - return defaults[name] + return value updated = _DOCKER_VARIABLE_RE.sub(_replace, resolved) resolved = updated @@ -2404,7 +2414,7 @@ def _apply_env(payload: str) -> None: raise ValueError(f"Cannot safely parse custom Dockerfile ENV: {payload}") from exc pairs: list[tuple[str, str]] = [] if assignments and all("=" in assignment for assignment in assignments): - pairs = [tuple(assignment.split("=", 1)) for assignment in assignments] + pairs = [(key, val) for key, _, val in (assignment.partition("=") for assignment in assignments)] elif len(assignments) >= 2: pairs = [(assignments[0], " ".join(assignments[1:]))] if not pairs: @@ -4307,15 +4317,27 @@ def _native_task_workdir(task_dir: Path, *, allow_docker_image: bool = False) -> return _validated_agent_workdir(workdir) +def _toml_table_header(line: str) -> str | None: + """Return the normalized '[...]' or '[[...]]' header from a TOML line, or None.""" + stripped = line.split("#", 1)[0].strip() + if stripped.startswith("[") and stripped.endswith("]"): + return stripped + return None + + def _ensure_native_skills_dir(task_dir: Path) -> None: """Pin native tasks to the evaluator-owned runtime skill projection.""" task_toml = task_dir / "task.toml" content = task_toml.read_text(encoding="utf-8") lines = content.splitlines() - if "[environment]" in lines: - environment_index = lines.index("[environment]") + environment_index = next((i for i, line in enumerate(lines) if _toml_table_header(line) == "[environment]"), None) + if environment_index is not None: section_end = next( - (index for index in range(environment_index + 1, len(lines)) if lines[index].strip().startswith("[")), + ( + index + for index in range(environment_index + 1, len(lines)) + if _toml_table_header(lines[index]) is not None + ), len(lines), ) if any(line.strip().startswith("skills_dir") for line in lines[environment_index + 1 : section_end]): @@ -4326,38 +4348,76 @@ def _ensure_native_skills_dir(task_dir: Path) -> None: task_toml.write_text("\n".join(lines) + "\n", encoding="utf-8") +def _merge_toml_env_table_section( + lines: list[str], + header_idx: int, + rendered: dict[str, str], + *, + prepend_missing: bool = False, +) -> None: + """Replace colliding keys in-place inside a TOML env table section and add missing keys.""" + idx = header_idx + 1 + env_end = next( + (end_idx for end_idx in range(idx, len(lines)) if _toml_table_header(lines[end_idx]) is not None), + len(lines), + ) + seen_keys: set[str] = set() + updated_section: list[str] = [] + for line in lines[idx:env_end]: + stripped_line = line.strip() + assignment = ( + line.split("=", 1)[0].strip() if stripped_line and not stripped_line.startswith("#") and "=" in line else "" + ) + matching_key = next( + (key for key in rendered if assignment in {key, _toml_quote(key), f"'{key}'"}), + None, + ) + if matching_key is not None: + if matching_key in seen_keys: + continue + updated_section.append(rendered[matching_key]) + seen_keys.add(matching_key) + else: + updated_section.append(line) + missing_lines = [line for key, line in rendered.items() if key not in seen_keys] + if missing_lines: + if prepend_missing: + updated_section[0:0] = missing_lines + else: + updated_section.extend(missing_lines) + lines[idx:env_end] = updated_section + + def _ensure_skill_evaluator_verifier_env(task_dir: Path, *, verifier_env: dict[str, str] | None) -> None: """Ensure staged native tasks forward configured public provider variables.""" + rendered = {name: f'{name} = "${{{name}}}"' for name in _verifier_env_vars(verifier_env)} + if not rendered: + return task_toml = task_dir / "task.toml" content = task_toml.read_text(encoding="utf-8") - env_lines = [f'{name} = "${{{name}}}"' for name in _verifier_env_vars(verifier_env)] - if all(line in content for line in env_lines): - return + env_lines = list(rendered.values()) lines = content.splitlines() - if "[verifier.env]" in lines: - idx = lines.index("[verifier.env]") + 1 - existing = set(lines) - for line in reversed(env_lines): - if line not in existing: - lines.insert(idx, line) + verifier_env_idx = next((i for i, line in enumerate(lines) if _toml_table_header(line) == "[verifier.env]"), None) + if verifier_env_idx is not None: + _merge_toml_env_table_section(lines, verifier_env_idx, rendered, prepend_missing=True) task_toml.write_text("\n".join(lines) + "\n", encoding="utf-8") return env_block = ["[verifier.env]", *env_lines] - if "[verifier]" in lines: - start = lines.index("[verifier]") + 1 - insert_at = len(lines) - for idx in range(start, len(lines)): - line = lines[idx].strip() - if line.startswith("[") and line.endswith("]"): - insert_at = idx - break + verifier_idx = next((i for i, line in enumerate(lines) if _toml_table_header(line) == "[verifier]"), None) + if verifier_idx is not None: + start = verifier_idx + 1 + insert_at = next( + (idx for idx in range(start, len(lines)) if _toml_table_header(lines[idx]) is not None), + len(lines), + ) lines[insert_at:insert_at] = ["", *env_block] task_toml.write_text("\n".join(lines) + "\n", encoding="utf-8") return - insert_at = lines.index("[environment]") if "[environment]" in lines else len(lines) + env_table_idx = next((i for i, line in enumerate(lines) if _toml_table_header(line) == "[environment]"), None) + insert_at = env_table_idx if env_table_idx is not None else len(lines) lines[insert_at:insert_at] = [ "[verifier]", f"timeout_sec = {DEFAULT_LLM_VERIFIER_TIMEOUT_SEC}", @@ -4370,20 +4430,21 @@ def _ensure_skill_evaluator_verifier_env(task_dir: Path, *, verifier_env: dict[s def _insert_table_block(lines: list[str], anchor: str, block: list[str]) -> None: """Insert a TOML table block after *anchor* and before the next table.""" - if anchor not in lines: + anchor_idx = next((i for i, line in enumerate(lines) if _toml_table_header(line) == anchor), None) + if anchor_idx is None: lines.extend(["", anchor]) - start = lines.index(anchor) + 1 - insert_at = len(lines) - for idx in range(start, len(lines)): - stripped = lines[idx].strip() - if stripped.startswith("[") and stripped.endswith("]"): - insert_at = idx - break + anchor_idx = len(lines) - 1 + start = anchor_idx + 1 + insert_at = next( + (idx for idx in range(start, len(lines)) if _toml_table_header(lines[idx]) is not None), + len(lines), + ) prefix = [] if insert_at == 0 or (insert_at > 0 and lines[insert_at - 1] == "") else [""] lines[insert_at:insert_at] = [*prefix, *block] def _ensure_environment_env(task_dir: Path, runtime_env: dict[str, str]) -> None: + """Ensure staged native tasks populate [environment.env] with runtime environment variables.""" runtime_env = {**runtime_env, **_EVALUATOR_MANAGED_RUNTIME_ENV} task_toml = task_dir / "task.toml" @@ -4391,33 +4452,9 @@ def _ensure_environment_env(task_dir: Path, runtime_env: dict[str, str]) -> None header = "[environment.env]" rendered = {key: f"{_toml_quote(key)} = {_toml_quote(runtime_env[key])}" for key in sorted(runtime_env)} env_lines = list(rendered.values()) - if header in lines: - idx = lines.index(header) + 1 - env_end = len(lines) - for end_idx in range(idx, len(lines)): - stripped = lines[end_idx].strip() - if stripped.startswith("[") and stripped.endswith("]"): - env_end = end_idx - break - seen_keys: set[str] = set() - updated_section: list[str] = [] - for line in lines[idx:env_end]: - assignment = line.split("=", 1)[0].strip() if line.strip() and "=" in line else "" - matching_key = next( - (key for key in runtime_env if assignment in {key, _toml_quote(key)}), - None, - ) - if matching_key is not None: - if matching_key in seen_keys: - continue - updated_section.append(rendered[matching_key]) - seen_keys.add(matching_key) - else: - updated_section.append(line) - for key, line in rendered.items(): - if key not in seen_keys: - updated_section.append(line) - lines[idx:env_end] = updated_section + header_idx = next((i for i, line in enumerate(lines) if _toml_table_header(line) == header), None) + if header_idx is not None: + _merge_toml_env_table_section(lines, header_idx, rendered, prepend_missing=False) task_toml.write_text("\n".join(lines) + "\n", encoding="utf-8") return diff --git a/src/skillevaluator/tier3/harbor/runner.py b/src/skillevaluator/tier3/harbor/runner.py index 1da1ecd91..cf3de5f33 100644 --- a/src/skillevaluator/tier3/harbor/runner.py +++ b/src/skillevaluator/tier3/harbor/runner.py @@ -44,6 +44,7 @@ from skillevaluator.tier3.harbor.adapter import ( _VERIFIER_BUDGET_ENV_VARS, _VERIFIER_JUDGE_MODEL_ENV_VARS, + _VERIFIER_RETRY_ENV_VARS, _prevalidate_baseline_skill_candidates, build_eval_base_image, find_evals_file, @@ -483,7 +484,7 @@ def _provider_environment(config: ProviderConfig) -> dict[str, str]: environment.update( { name: value - for name in (_VERIFIER_JUDGE_MODEL_ENV_VARS | _VERIFIER_BUDGET_ENV_VARS) + for name in (_VERIFIER_JUDGE_MODEL_ENV_VARS | _VERIFIER_BUDGET_ENV_VARS | _VERIFIER_RETRY_ENV_VARS) if (value := os.environ.get(name, "").strip()) } ) diff --git a/src/skillevaluator/tier3/harbor/templates/eval.py b/src/skillevaluator/tier3/harbor/templates/eval.py index 1a4d949f2..9128cfdf9 100644 --- a/src/skillevaluator/tier3/harbor/templates/eval.py +++ b/src/skillevaluator/tier3/harbor/templates/eval.py @@ -22,21 +22,28 @@ from __future__ import annotations +import contextlib +import io import ipaddress import json import logging import math import os +import random import re import shlex +import signal import sys +import threading +import time import unicodedata import urllib.error import urllib.request +from contextvars import ContextVar from fnmatch import fnmatchcase from functools import lru_cache from pathlib import Path -from typing import Any +from typing import Any, NamedTuple from urllib.parse import quote, unquote_to_bytes, urlparse, urlsplit import idna @@ -641,9 +648,7 @@ def build_behavior_evidence( ): """Build compact, behavior-check-specific evidence from an ATIF trajectory.""" effective_final_limit = ( - _behavior_final_response_limit() - if final_response_limit is None - else max(1, int(final_response_limit)) + _behavior_final_response_limit() if final_response_limit is None else max(1, int(final_response_limit)) ) if max_chars is None: @@ -665,9 +670,7 @@ def build_behavior_evidence( user_needed = ( min(800, len(_SECTION_USER_REQUEST) + len(question.strip()) + 2) if question and question.strip() else 0 ) - history_needed = ( - len(_SECTION_COMPACT_TOOL_HISTORY) + len(history.strip()) + 2 if history and history.strip() else 0 - ) + history_needed = len(_SECTION_COMPACT_TOOL_HISTORY) + len(history.strip()) + 2 if history and history.strip() else 0 tail_needed = user_needed + (2 if user_needed and history_needed else 0) + history_needed if file_changes: @@ -1543,7 +1546,36 @@ def _is_native_openai_chat_url(provider, request_url): ) -def _chat_completion_payload(model, prompt, max_tokens, temperature, provider=None, request_url=None): +def _build_openai_response_format(schema, schema_name="judge_response"): + return { + "type": "json_schema", + "json_schema": { + "name": schema_name, + "strict": True, + "schema": schema, + }, + } + + +def _build_anthropic_output_config(schema): + return { + "format": { + "type": "json_schema", + "schema": schema, + } + } + + +def _chat_completion_payload( + model, + prompt, + max_tokens, + temperature, + provider=None, + request_url=None, + response_schema=None, + schema_name="judge_response", +): resolved_provider = _public_provider() if provider is None else provider resolved_request_url = _resolve_url(resolved_provider) if request_url is None else request_url token_key = ( @@ -1560,6 +1592,8 @@ def _chat_completion_payload(model, prompt, max_tokens, temperature, provider=No } if temperature is not None and _supports_custom_temperature(model): payload["temperature"] = temperature + if response_schema is not None: + payload["response_format"] = _build_openai_response_format(response_schema, schema_name) return payload @@ -1746,30 +1780,310 @@ def _anthropic_url(): return _validate_http_url(url) -def _call_anthropic(prompt, model, max_tokens, temperature): +_RETRIABLE_HTTP_CODES = frozenset({429, 500, 502, 503, 504}) +_DEFAULT_MAX_RETRIES = 3 +_DEFAULT_BASE_DELAY = 1.0 +_DEFAULT_MAX_DELAY = 30.0 +# Three required judges run sequentially under the managed 600-second Harbor +# verifier timeout. Reserve one minute for deterministic checks and artifacts. +_JUDGE_WALL_TIME_BUDGET_SEC = 180.0 +_ACTIVE_JUDGE_DEADLINE: ContextVar[float | None] = ContextVar("active_judge_deadline", default=None) + + +class EvalRetryConfig(NamedTuple): + """Represent bounded retry and backoff settings for direct verifier LLM calls.""" + + max_retries: int + base_delay: float + max_delay: float + + +class SchemaTargetKey(NamedTuple): + """Identify a provider endpoint and model for structured output schema memoization.""" + + provider: str + base_url: str + model: str + + +def _resolve_judge_wall_time_budget(): + """Resolve the per-judge wall-time budget in seconds from the environment.""" + raw = str(os.environ.get("SKILL_EVAL_LLM_JUDGE_BUDGET_SEC", "")).strip() + if raw: + try: + val = float(raw) + if math.isfinite(val) and val > 0.0: + return val + except ValueError: + pass + return _JUDGE_WALL_TIME_BUDGET_SEC + + +def _remaining_judge_timeout(timeout): + """Bound one provider request by the remaining time for its judge.""" + deadline = _ACTIVE_JUDGE_DEADLINE.get() + if deadline is None: + return timeout + remaining = deadline - time.monotonic() + if remaining <= 0: + raise TimeoutError("LLM judge time budget exhausted") + return min(timeout, remaining) + + +def _parse_retry_after(header_value, fallback_delay): + """Parse a Retry-After header as seconds or HTTP date, falling back to default.""" + if not header_value: + return fallback_delay + clean_val = str(header_value).strip() + try: + return max(0.0, float(clean_val)) + except ValueError: + pass + try: + from datetime import UTC, datetime + from email.utils import parsedate_to_datetime + + target = parsedate_to_datetime(clean_val) + if target.tzinfo is None: + target = target.replace(tzinfo=UTC) + now = datetime.now(UTC) + return max(0.0, (target - now).total_seconds()) + except Exception: + return fallback_delay + + +def _calculate_jitter_delay(attempt, base_delay=1.0, max_delay=30.0): + """Calculate exponential backoff with full jitter.""" + calculated = min(max_delay, base_delay * (2.0**attempt)) + return random.uniform(0.0, calculated) + + +def _resolve_eval_retry_config(): + """Resolve retry and backoff limits from environment variables with safe defaults.""" + + def _read_int(name, default): + """Read a non-negative integer from the environment variable or return default.""" + raw = str(os.environ.get(name, "")).strip() + if raw: + try: + val = int(raw) + return val if val >= 0 else default + except ValueError: + return default + return default + + def _read_float(name, default): + """Read a non-negative float from the environment variable or return default.""" + raw = str(os.environ.get(name, "")).strip() + if raw: + try: + val = float(raw) + return val if math.isfinite(val) and val >= 0.0 else default + except ValueError: + return default + return default + + max_retries = _read_int("SKILL_EVAL_LLM_MAX_RETRIES", _DEFAULT_MAX_RETRIES) + base_delay = _read_float("SKILL_EVAL_LLM_RETRY_BASE_DELAY", _DEFAULT_BASE_DELAY) + raw_max_delay = _read_float("SKILL_EVAL_LLM_RETRY_MAX_DELAY", _DEFAULT_MAX_DELAY) + return EvalRetryConfig(max_retries=max_retries, base_delay=base_delay, max_delay=max(base_delay, raw_max_delay)) + + +def _compute_bounded_retry_delay(retry_after_str, *, attempt, base_delay, max_delay, error): + """Compute a bounded retry sleep duration and verify the judge deadline allows it.""" + if retry_after_str is not None: + parsed = _parse_retry_after(retry_after_str, fallback_delay=base_delay) + if parsed > max_delay: + raise error + delay = parsed + random.uniform(0.1, 0.5) + else: + delay = _calculate_jitter_delay(attempt, base_delay=base_delay, max_delay=max_delay) + + sleep_duration = min(delay, max_delay) + deadline = _ACTIVE_JUDGE_DEADLINE.get() + if deadline is not None and time.monotonic() + sleep_duration >= deadline: + raise TimeoutError("LLM judge time budget exhausted before retry") from error + return sleep_duration + + +def _urlopen_with_retry(request, timeout=90): + """Open a URL request with exponential backoff and full jitter on transient failures.""" + retry_config = _resolve_eval_retry_config() + attempt = 0 + while True: + request_timeout = _remaining_judge_timeout(timeout) + try: + with urllib.request.urlopen(request, timeout=request_timeout) as response: # nosec B310 + return response.read() + except Exception as error: + is_http = isinstance(error, urllib.error.HTTPError) + is_network = isinstance(error, (urllib.error.URLError, ConnectionError, OSError)) + if ( + attempt >= retry_config.max_retries + or (not is_http and not is_network) + or (is_http and error.code not in _RETRIABLE_HTTP_CODES) + ): + raise + + retry_after_str = None + if is_http and error.headers: + retry_after_str = error.headers.get("retry-after") or error.headers.get("Retry-After") + + sleep_duration = _compute_bounded_retry_delay( + retry_after_str, + attempt=attempt, + base_delay=retry_config.base_delay, + max_delay=retry_config.max_delay, + error=error, + ) + status_label = f"HTTP {error.code}" if is_http else type(error).__name__ + logger.warning( + "LLM judge transient error (%s). Retrying in %.2fs (attempt %d/%d)...", + status_label, + sleep_duration, + attempt + 1, + retry_config.max_retries, + ) + if is_http: + error.close() + time.sleep(sleep_duration) + attempt += 1 + + +_UNSUPPORTED_REASON_INDICATORS = ( + "unsupported", + "not supported", + "extra input", + "extra inputs", + "unknown parameter", + "unknown field", + "unknown argument", + "unrecognized request argument", + "unrecognized parameter", + "unexpected keyword argument", + "unexpected argument", + "invalid parameter", + "invalid argument", + "not permitted", + "not allowed", + "disallowed", +) + +_SCHEMA_OPTION_PATTERN = r"(?:response_format|response format|output_config|json_schema|structured[_ ]outputs?)" +_SCHEMA_REJECTION_REASON = ( + r"(?:unsupported|not supported|not permitted|not allowed|disallowed|" + r"unknown (?:parameter|field|argument)|unrecognized (?:request argument|parameter)|" + r"unexpected (?:keyword )?argument|extra inputs?(?: are not permitted)?)" +) +_SCHEMA_REJECTION_AFTER_OPTION = re.compile( + rf"\b{_SCHEMA_OPTION_PATTERN}\b(?:\.[a-z0-9_]+)*" + rf"(?:\s+of\s+type\s+['\"]?[a-z0-9_]+['\"]?)?" + rf"\s*(?:(?:is|are|was|were)\s+(?:an?\s+)?|:\s*)?" + rf"{_SCHEMA_REJECTION_REASON}\b", + re.IGNORECASE, +) +_SCHEMA_REJECTION_BEFORE_OPTION = re.compile( + rf"\b(?:unsupported|not supported|extra inputs?(?: are not permitted)?|unknown (?:parameter|field|argument)|" + rf"unrecognized (?:request argument|parameter)|unexpected (?:keyword argument|argument)|" + rf"invalid (?:parameter|argument)|not permitted|not allowed|disallowed)\b" + rf"(?:\s+supplied)?[\s:'\"\[\]{{}}(),-]{{0,32}}\b{_SCHEMA_OPTION_PATTERN}\b", + re.IGNORECASE, +) + + +def _message_rejects_schema_option(text, param=None): + """Match a rejection of the schema option itself, not unrelated error text.""" + if param: + if not re.search(rf"\b{_SCHEMA_OPTION_PATTERN}\b", param, re.IGNORECASE): + return False + return any(indicator in text.lower() for indicator in _UNSUPPORTED_REASON_INDICATORS) + return bool(_SCHEMA_REJECTION_AFTER_OPTION.search(text) or _SCHEMA_REJECTION_BEFORE_OPTION.search(text)) + + +def _is_schema_unsupported_http_error(error): + """Determine whether an HTTP error indicates structured output schema is unsupported.""" + if getattr(error, "code", None) not in {400, 422}: + return False + body_text = "" + try: + body_bytes = error.read() + error.fp = io.BytesIO(body_bytes) + body_text = body_bytes.decode("utf-8", "replace") + except Exception: + pass + error_param = None + try: + body = json.loads(body_text) + if isinstance(body, dict) and isinstance(body.get("error"), dict): + param = body["error"].get("param") + if isinstance(param, str): + error_param = param + except (TypeError, ValueError): + pass + text = f"{error} {getattr(error, 'reason', '')} {body_text}" + return _message_rejects_schema_option(text, error_param) + + +_SCHEMA_UNSUPPORTED_TARGETS: set[SchemaTargetKey] = set() + + +def _urlopen_with_schema_fallback(build_request, *, target_key, use_schema, timeout=90): + """Open URL with retry, falling back to prompt-only on confirmed schema capability errors.""" + normalized_key = target_key if isinstance(target_key, SchemaTargetKey) else SchemaTargetKey(*target_key) + try: + return _urlopen_with_retry(build_request(use_schema), timeout=timeout) + except urllib.error.HTTPError as error: + if use_schema and _is_schema_unsupported_http_error(error): + error.close() + logger.warning( + "Structured output schema unsupported by provider=%s model=%s; " + "downgrading to prompt-only JSON and memoizing target.", + normalized_key.provider, + normalized_key.model, + ) + response = _urlopen_with_retry(build_request(False), timeout=timeout) + _SCHEMA_UNSUPPORTED_TARGETS.add(normalized_key) + return response + raise + + +def _call_anthropic(prompt, model, max_tokens, temperature, response_schema=None): api_key = os.environ.get("ANTHROPIC_API_KEY", "") if not api_key: return None, "ANTHROPIC_API_KEY is required for the anthropic provider" - payload = { - "model": model, - "max_tokens": max_tokens, - "messages": [{"role": "user", "content": prompt}], - "stream": False, - } - if temperature is not None and _supports_custom_temperature(model): - payload["temperature"] = temperature - request = urllib.request.Request( - _anthropic_url(), - data=json.dumps(payload).encode(), - headers={ - "Content-Type": "application/json", - "x-api-key": api_key, - "anthropic-version": "2023-06-01", - }, - ) + target_url = _anthropic_url() + target_key = SchemaTargetKey(provider="anthropic", base_url=target_url, model=model) + use_schema = response_schema is not None and target_key not in _SCHEMA_UNSUPPORTED_TARGETS + + def _build_request(include_schema): + payload = { + "model": model, + "max_tokens": max_tokens, + "messages": [{"role": "user", "content": prompt}], + "stream": False, + } + if temperature is not None and _supports_custom_temperature(model): + payload["temperature"] = temperature + if include_schema: + payload["output_config"] = _build_anthropic_output_config(response_schema) + return urllib.request.Request( + target_url, + data=json.dumps(payload).encode(), + headers={ + "Content-Type": "application/json", + "x-api-key": api_key, + "anthropic-version": "2023-06-01", + }, + ) + # _anthropic_url() validates the configured base URL before this request. - with urllib.request.urlopen(request, timeout=90) as response: # nosec B310 - body = json.loads(response.read()) + raw_response = _urlopen_with_schema_fallback( + _build_request, + target_key=target_key, + use_schema=use_schema, + timeout=90, + ) + body = json.loads(raw_response) content = "".join( str(block.get("text", "")) for block in body.get("content", []) @@ -1778,21 +2092,138 @@ def _call_anthropic(prompt, model, max_tokens, temperature): return content.strip(), None -def _call_bedrock(prompt, model, max_tokens, temperature): +_RETRIABLE_BEDROCK_ERROR_CODES = frozenset( + { + "ThrottlingException", + "Throttling", + "TooManyRequestsException", + "ServiceUnavailableException", + "ServiceUnavailable", + "InternalServerException", + "InternalServerError", + "InternalFailure", + "ModelTimeoutException", + "RequestTimeout", + "RequestTimeoutException", + } +) +_RETRIABLE_BOTOCORE_EXCEPTION_NAMES = frozenset( + { + "EndpointConnectionError", + "ConnectionClosedError", + "ReadTimeoutError", + "ConnectTimeoutError", + } +) + + +def _classify_bedrock_retry_error(error): + """Return (is_retriable, status_label, retry_after_str) for a Bedrock Converse exception.""" + if isinstance(error, TimeoutError) and "LLM judge time budget exhausted" in str(error): + return False, type(error).__name__, None + if isinstance(error, (FileNotFoundError, IsADirectoryError, NotADirectoryError, PermissionError)): + return False, type(error).__name__, None + + response = getattr(error, "response", None) + if not isinstance(response, dict): + response = {} + metadata = response.get("ResponseMetadata") + if not isinstance(metadata, dict): + metadata = {} + http_status = metadata.get("HTTPStatusCode") + if not isinstance(http_status, int) or isinstance(http_status, bool): + http_status = None + + err_info = response.get("Error") + if not isinstance(err_info, dict): + err_info = {} + error_code = str(err_info.get("Code") or "").strip() + + headers = metadata.get("HTTPHeaders") + retry_after_str = None + if isinstance(headers, dict): + for k, v in headers.items(): + if isinstance(k, str) and k.lower() == "retry-after" and v is not None: + retry_after_str = str(v) + break + + if http_status is not None or error_code: + is_retriable = ( + http_status in _RETRIABLE_HTTP_CODES or http_status == 408 or error_code in _RETRIABLE_BEDROCK_ERROR_CODES + ) + status_label = f"HTTP {http_status}" if http_status is not None else error_code + return is_retriable, status_label, retry_after_str + + type_name = type(error).__name__ + is_network = ( + isinstance(error, (ConnectionError, TimeoutError, OSError)) or type_name in _RETRIABLE_BOTOCORE_EXCEPTION_NAMES + ) + return is_network, type_name, None + + +def _call_bedrock(prompt, model, max_tokens, temperature, timeout=90): try: import boto3 except ImportError: return None, "boto3 is required for the bedrock provider" + BotoConfig = None + with contextlib.suppress(ImportError): + from botocore.config import Config as BotoConfig try: - client = boto3.client("bedrock-runtime", region_name=os.environ.get("AWS_REGION", "us-west-2")) + retry_config = _resolve_eval_retry_config() + region_name = os.environ.get("AWS_REGION", "us-west-2") + initial_timeout = _remaining_judge_timeout(timeout) + client_kwargs = {"region_name": region_name} + if BotoConfig is not None: + client_kwargs["config"] = BotoConfig( + connect_timeout=initial_timeout, + read_timeout=initial_timeout, + retries={"max_attempts": 0, "mode": "standard"}, + ) + try: + client = boto3.client("bedrock-runtime", **client_kwargs) + except TypeError: + client = boto3.client("bedrock-runtime", region_name=region_name) + inference_config = {"maxTokens": max_tokens} if temperature is not None and _supports_custom_temperature(model): inference_config["temperature"] = temperature - response = client.converse( - modelId=model, - messages=[{"role": "user", "content": [{"text": prompt}]}], - inferenceConfig=inference_config, - ) + + attempt = 0 + while True: + request_timeout = _remaining_judge_timeout(timeout) + endpoint = getattr(client, "_endpoint", None) + if endpoint is not None and hasattr(endpoint, "timeout"): + endpoint.timeout = request_timeout + try: + response = client.converse( + modelId=model, + messages=[{"role": "user", "content": [{"text": prompt}]}], + inferenceConfig=inference_config, + ) + break + except Exception as error: + is_retriable, status_label, retry_after_str = _classify_bedrock_retry_error(error) + if attempt >= retry_config.max_retries or not is_retriable: + raise + + sleep_duration = _compute_bounded_retry_delay( + retry_after_str, + attempt=attempt, + base_delay=retry_config.base_delay, + max_delay=retry_config.max_delay, + error=error, + ) + logger.warning( + "LLM judge transient error (%s). Retrying in %.2fs (attempt %d/%d)...", + status_label, + sleep_duration, + attempt + 1, + retry_config.max_retries, + ) + time.sleep(sleep_duration) + attempt += 1 + content = "".join( str(block.get("text", "")) for block in response.get("output", {}).get("message", {}).get("content", []) @@ -1813,7 +2244,15 @@ def _selected_judge_model(model=None): ) -def _call_public_llm_with_provenance(prompt, model=None, max_tokens=1024, temperature=0.0, allow_model_fallback=True): +def _call_public_llm_with_provenance( + prompt, + model=None, + max_tokens=1024, + temperature=0.0, + allow_model_fallback=True, + response_schema=None, + schema_name="judge_response", +): provider = _public_provider() if not provider: return None, _public_provider_error(), {} @@ -1826,7 +2265,13 @@ def _call_public_llm_with_provenance(prompt, model=None, max_tokens=1024, temper last_provenance = provenance try: if provider == "anthropic": - content, error = _call_anthropic(prompt, candidate_model, max_tokens, temperature) + content, error = _call_anthropic( + prompt, + candidate_model, + max_tokens, + temperature, + response_schema=response_schema, + ) if error: return None, _redact_configured_credentials(error), provenance return content, None, provenance @@ -1844,24 +2289,45 @@ def _call_public_llm_with_provenance(prompt, model=None, max_tokens=1024, temper if not api_key: return None, f"No API key configured for {provider}", provenance request_url = _resolve_url(provider) - request = urllib.request.Request( - request_url, - data=json.dumps( - _chat_completion_payload( - candidate_model, - prompt, - max_tokens, - temperature, - provider=provider, - request_url=request_url, - ) - ).encode(), - headers={"Content-Type": "application/json", "Authorization": f"Bearer {api_key}"}, - ) + target_key = SchemaTargetKey(provider=provider, base_url=request_url, model=candidate_model) + use_schema = response_schema is not None and target_key not in _SCHEMA_UNSUPPORTED_TARGETS + + def _build_oai_request( + include_schema, + *, + _url=request_url, + _model=candidate_model, + _key=api_key, + ): + return urllib.request.Request( + _url, + data=json.dumps( + _chat_completion_payload( + _model, + prompt, + max_tokens, + temperature, + provider=provider, + request_url=_url, + response_schema=response_schema if include_schema else None, + schema_name=schema_name, + ) + ).encode(), + headers={"Content-Type": "application/json", "Authorization": f"Bearer {_key}"}, + ) + # request_url was validated by _resolve_url() before this request. - with urllib.request.urlopen(request, timeout=90) as response: # nosec B310 - body = json.loads(response.read()) - content = body.get("choices", [{}])[0].get("message", {}).get("content", "") + raw_response = _urlopen_with_schema_fallback( + _build_oai_request, + target_key=target_key, + use_schema=use_schema, + timeout=90, + ) + body = json.loads(raw_response) + choices = body.get("choices") or [{}] + first_choice = choices[0] if choices and isinstance(choices[0], dict) else {} + message = first_choice.get("message") + content = message.get("content", "") if isinstance(message, dict) else "" if content is None: content = "" if candidate_model != requested_model: @@ -1879,13 +2345,23 @@ def _call_public_llm_with_provenance(prompt, model=None, max_tokens=1024, temper return None, _redact_configured_credentials(detail), last_provenance -def call_public_llm(prompt, model=None, max_tokens=1024, temperature=0.0, allow_model_fallback=True): +def call_public_llm( + prompt, + model=None, + max_tokens=1024, + temperature=0.0, + allow_model_fallback=True, + response_schema=None, + schema_name="judge_response", +): content, error, _provenance = _call_public_llm_with_provenance( prompt, model=model, max_tokens=max_tokens, temperature=temperature, allow_model_fallback=allow_model_fallback, + response_schema=response_schema, + schema_name=schema_name, ) return content, error @@ -7493,6 +7969,7 @@ def score_skill_execution( def _call_validated_json_judge(prompt, validate, call, extract, **call_kwargs): + """Invoke a JSON judge with one format-correction retry when payload validation fails.""" call_kwargs.setdefault("max_tokens", STRUCTURED_JUDGE_MAX_TOKENS) def invoke(call_prompt): @@ -7531,6 +8008,7 @@ def invoke(call_prompt): def _valid_accuracy_criteria(value): + """Return True when value is a complete 5-criterion boolean mapping.""" return ( isinstance(value, dict) and value.keys() == _ACCURACY_CRITERIA_KEYS @@ -7539,6 +8017,7 @@ def _valid_accuracy_criteria(value): def _accuracy_payload_error(parsed): + """Validate a parsed accuracy judge payload and return an error message if malformed.""" if not isinstance(parsed, dict): return "Judge response was not a valid JSON object" if "reason" in parsed and not isinstance(parsed["reason"], str): @@ -7553,6 +8032,7 @@ def _accuracy_payload_error(parsed): def _goal_payload_error(parsed): + """Validate a parsed goal-accuracy judge payload and return an error message if malformed.""" if not isinstance(parsed, dict): return "Judge response was not a valid JSON object" for field in ("reason", "user_goal", "end_state"): @@ -7565,11 +8045,76 @@ def _goal_payload_error(parsed): return None +ACCURACY_JSON_SCHEMA = { + "type": "object", + "properties": { + "criteria": { + "type": "object", + "properties": { + "SKILL_IDENTIFIED": {"type": "boolean"}, + "ACTION_CORRECT": {"type": "boolean"}, + "FACTUALLY_ACCURATE": {"type": "boolean"}, + "TASK_ADDRESSED": {"type": "boolean"}, + "ACTIONABLE": {"type": "boolean"}, + }, + "required": [ + "SKILL_IDENTIFIED", + "ACTION_CORRECT", + "FACTUALLY_ACCURATE", + "TASK_ADDRESSED", + "ACTIONABLE", + ], + "additionalProperties": False, + }, + "score": {"type": "number"}, + "reason": {"type": "string"}, + }, + "required": ["criteria", "score", "reason"], + "additionalProperties": False, +} + +GOAL_ACCURACY_JSON_SCHEMA = { + "type": "object", + "properties": { + "user_goal": {"type": "string"}, + "end_state": {"type": "string"}, + "achieved": {"type": "boolean"}, + "score": {"type": "number"}, + "reason": {"type": "string"}, + }, + "required": ["user_goal", "end_state", "achieved", "score", "reason"], + "additionalProperties": False, +} + +BEHAVIOR_CHECK_JSON_SCHEMA = { + "type": "object", + "properties": { + "results": { + "type": "array", + "items": { + "type": "object", + "properties": { + "step": {"type": "integer"}, + "passed": {"type": "boolean"}, + "reason": {"type": "string"}, + }, + "required": ["step", "passed", "reason"], + "additionalProperties": False, + }, + }, + "score": {"type": "number"}, + "summary": {"type": "string"}, + }, + "required": ["results", "score", "summary"], + "additionalProperties": False, +} + + def judge_accuracy(question, ground_truth, agent_text): if not ground_truth: return {"score": 1.0, "reason": "No ground_truth -- skipped"} prompt = f"""You are an expert evaluator for AI agent responses. Evaluate by checking \ -each criterion below against the expected answer. For each, answer YES or NO. +each criterion below against the expected answer. For each criterion, determine true (satisfied) or false (not satisfied). 1. SKILL_IDENTIFIED: Does the response reference or use the correct skill for the task? 2. ACTION_CORRECT: Does the response describe or execute the correct actions/scripts? @@ -7577,8 +8122,7 @@ def judge_accuracy(question, ground_truth, agent_text): 4. TASK_ADDRESSED: Does the response directly address the user's request? 5. ACTIONABLE: Does the response provide actionable information (not just acknowledgment)? -For each criterion write: YES or NO with a brief reason. -Then compute score = count(YES) / 5. +Compute score = count(true) / 5. Be lenient on exact wording but strict on factual correctness. Respond with ONLY a JSON object: @@ -7598,6 +8142,8 @@ def judge_accuracy(question, ground_truth, agent_text): _accuracy_payload_error, call_public_llm, extract_json, + response_schema=ACCURACY_JSON_SCHEMA, + schema_name="accuracy_judgment", ) if error: return _judge_error(error) @@ -7680,7 +8226,12 @@ def _judge_goal_accuracy_ragas(question, ground_truth, agent_text, tool_summary) loop = asyncio.new_event_loop() try: - result = loop.run_until_complete(metric.ascore(sample)) + result = loop.run_until_complete( + asyncio.wait_for( + metric.ascore(sample), + timeout=_remaining_judge_timeout(_resolve_judge_wall_time_budget()), + ) + ) finally: loop.close() @@ -7725,6 +8276,8 @@ def _judge_goal_accuracy_custom(question, ground_truth, agent_text, tool_summary _goal_payload_error, _call_public_llm_with_provenance, extract_json, + response_schema=GOAL_ACCURACY_JSON_SCHEMA, + schema_name="goal_accuracy_judgment", ) if error: return _judge_error(error, **provenance) @@ -7777,12 +8330,17 @@ def judge_behavior_check(conversation_text, expected_behaviors): EXPECTED BEHAVIORS: {behaviors_text} -For each behavior, respond YES (observed) or NO (not observed) with a brief reason. +For each behavior, set "passed" to true (observed) or false (not observed) with a brief reason. Respond with ONLY a JSON object: {{"results": [{{"step": 1, "passed": true, "reason": "..."}}, ...], "score": 0.67, "summary": "brief summary"}}""" - content, error = call_public_llm(prompt, max_tokens=BEHAVIOR_JUDGE_MAX_TOKENS) + content, error = call_public_llm( + prompt, + max_tokens=BEHAVIOR_JUDGE_MAX_TOKENS, + response_schema=BEHAVIOR_CHECK_JSON_SCHEMA, + schema_name="behavior_check_judgment", + ) if error: return _judge_error(f"LLM judge error: {error}", results=[]) @@ -7796,7 +8354,10 @@ def _parse_judge_object(text): if score is None: # One retry max, with an explicit machine-readable-output reminder. retry_content, retry_error = call_public_llm( - prompt + _BEHAVIOR_RETRY_REMINDER, max_tokens=BEHAVIOR_JUDGE_MAX_TOKENS + prompt + _BEHAVIOR_RETRY_REMINDER, + max_tokens=BEHAVIOR_JUDGE_MAX_TOKENS, + response_schema=BEHAVIOR_CHECK_JSON_SCHEMA, + schema_name="behavior_check_judgment", ) if not retry_error: parsed = _parse_judge_object(retry_content) @@ -7896,10 +8457,44 @@ def _normalize_required_judge_result(metric, result): def _call_required_judge(metric, judge, *args, **kwargs): + """Run a required LLM judge under a bounded wall-time deadline and normalize its result.""" + previous_deadline = _ACTIVE_JUDGE_DEADLINE.get() + own_deadline = time.monotonic() + _resolve_judge_wall_time_budget() + deadline = min(previous_deadline, own_deadline) if previous_deadline is not None else own_deadline + token = _ACTIVE_JUDGE_DEADLINE.set(deadline) + alarm_armed = False + previous_alarm_handler = None + + # The standalone Harbor verifier runs judges on its main thread. Its + # interval timer also interrupts a response body that keeps trickling data + # inside one socket read, where urllib's idle timeout cannot help. try: - result = judge(*args, **kwargs) + try: + if hasattr(signal, "setitimer") and threading.current_thread() is threading.main_thread(): + active_alarm, repeat_interval = signal.getitimer(signal.ITIMER_REAL) + if active_alarm == 0 and repeat_interval == 0: + previous_alarm_handler = signal.getsignal(signal.SIGALRM) + + def _raise_judge_timeout(_signum, _frame): + raise TimeoutError("LLM judge time budget exhausted") + + signal.signal(signal.SIGALRM, _raise_judge_timeout) + try: + signal.setitimer(signal.ITIMER_REAL, max(deadline - time.monotonic(), 1e-6)) + alarm_armed = True + except OSError: + signal.signal(signal.SIGALRM, previous_alarm_handler) + result = judge(*args, **kwargs) + finally: + if alarm_armed: + try: + signal.setitimer(signal.ITIMER_REAL, 0) + finally: + signal.signal(signal.SIGALRM, previous_alarm_handler) except Exception as exc: result = _judge_error(f"Required {metric} judge raised {type(exc).__name__}: {exc}") + finally: + _ACTIVE_JUDGE_DEADLINE.reset(token) return _normalize_required_judge_result(metric, result) @@ -8073,7 +8668,7 @@ def main(): details[_m]["omitted"] = _b["omitted"] # ── Write results ──────────────────────────────────────────────────── - result = { + result: dict[str, Any] = { "security": security_score, "skill_execution": se_score, "skill_efficiency": sef_score, diff --git a/tests/conftest.py b/tests/conftest.py index 0f184e6f7..dbd9fb863 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -6,7 +6,14 @@ Based on SkillEvaluator HOW_TO_CONTRIBUTE_SKILLS.md and HOW_TO_CONTRIBUTE_WORKFLOW_RULES.md specifications. """ +from __future__ import annotations + +import importlib.util +import json +from collections.abc import Callable from pathlib import Path +from types import ModuleType +from typing import Any import pytest @@ -533,3 +540,58 @@ def team_workflow_dir(tmp_path: Path) -> Path: """) return workflow_dir + + +# ============================================================================= +# HARBOR VERIFIER & URLLIB TEST HELPERS +# ============================================================================= + +_HARBOR_EVAL_TEMPLATE_PATH = ( + Path(__file__).resolve().parent.parent / "src/skillevaluator/tier3/harbor/templates/eval.py" +) + + +def load_harbor_eval_template(module_name: str = "harbor_eval_template") -> ModuleType: + """Load the standalone Harbor verifier template module dynamically.""" + spec = importlib.util.spec_from_file_location(module_name, _HARBOR_EVAL_TEMPLATE_PATH) + assert spec is not None and spec.loader is not None + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod + + +class MockUrllibResponse: + """Mock a urllib response context manager returning serialized JSON or raw bytes.""" + + def __init__(self, payload: Any) -> None: + """Initialize the mock response with a JSON-serializable object, string, or bytes.""" + if isinstance(payload, bytes): + self._body = payload + elif isinstance(payload, str): + self._body = payload.encode("utf-8") + else: + self._body = json.dumps(payload).encode("utf-8") + + def read(self) -> bytes: + """Return the response body bytes.""" + return self._body + + def __enter__(self) -> MockUrllibResponse: + """Enter the response context manager.""" + return self + + def __exit__(self, *args: object) -> bool: + """Exit the response context manager without suppressing exceptions.""" + return False + + +@pytest.fixture +def harbor_eval_template() -> ModuleType: + """Return a freshly loaded standalone Harbor verifier template module.""" + return load_harbor_eval_template() + + +@pytest.fixture +def make_urllib_json_response() -> Callable[[Any], MockUrllibResponse]: + """Return a factory that builds MockUrllibResponse instances from payloads.""" + return MockUrllibResponse diff --git a/tests/inference/test_llm_client.py b/tests/inference/test_llm_client.py index f5df8665d..9dcad7b9f 100644 --- a/tests/inference/test_llm_client.py +++ b/tests/inference/test_llm_client.py @@ -76,7 +76,7 @@ def test_openai_provider_uses_public_openai_endpoint(self, monkeypatch: pytest.M client = LLMClient() assert client._get_client() is mock_openai - mock_cls.assert_called_once_with(api_key="test-key", base_url="https://api.openai.com/v1") + mock_cls.assert_called_once_with(api_key="test-key", base_url="https://api.openai.com/v1", max_retries=0) def test_missing_api_key_raises(self, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") @@ -213,7 +213,7 @@ def test_nvidia_build_ambient_base_url_cannot_redirect_sdk(self, monkeypatch: py with patch("openai.OpenAI", return_value=mock_openai) as mock_cls: assert LLMClient()._get_client() is mock_openai - mock_cls.assert_called_once_with(api_key="test-key", base_url=PUBLIC_NVIDIA_BUILD_BASE_URL) + mock_cls.assert_called_once_with(api_key="test-key", base_url=PUBLIC_NVIDIA_BUILD_BASE_URL, max_retries=0) def test_accepted_canonical_openai_url_constructs_real_sdk_client(self) -> None: client = LLMClient( @@ -544,7 +544,7 @@ def test_openai_provider_custom_base_url_uses_max_tokens( with patch("openai.OpenAI", return_value=mock_openai) as mock_cls: LLMClient(max_tokens=512).completions("system", "user") - mock_cls.assert_called_once_with(api_key="test-key", base_url="https://example.test/v1") + mock_cls.assert_called_once_with(api_key="test-key", base_url="https://example.test/v1", max_retries=0) call_kwargs = mock_openai.chat.completions.create.call_args.kwargs assert call_kwargs["max_tokens"] == 512 assert "max_completion_tokens" not in call_kwargs diff --git a/tests/test_harbor_runner_environment.py b/tests/test_harbor_runner_environment.py index 5ef717b65..02f0c8c9a 100644 --- a/tests/test_harbor_runner_environment.py +++ b/tests/test_harbor_runner_environment.py @@ -245,6 +245,69 @@ def test_provider_environment_forwards_each_host_judge_model_override( assert environment.get(name) == expected +@pytest.mark.parametrize( + "name", + [ + "SKILL_EVAL_LLM_MAX_RETRIES", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY", + ], +) +@pytest.mark.parametrize("provider_name", ["openai", "openai-compatible", "anthropic", "bedrock", "nv_build"]) +@pytest.mark.parametrize( + ("configured", "expected"), + [ + ("0", "0"), + (" 0.25 ", "0.25"), + ("", None), + (" ", None), + ], +) +def test_provider_environment_forwards_each_host_retry_override( + monkeypatch: pytest.MonkeyPatch, + name: str, + provider_name: str, + configured: str, + expected: str | None, +) -> None: + """Verify _provider_environment forwards stripped host retry settings and omits blank values.""" + monkeypatch.setattr(runner.os, "environ", {name: configured}) + + environment = runner._provider_environment(_provider(provider_name)) + + assert environment.get(name) == expected + + +def test_resolve_agent_runtime_plan_retains_retry_settings_in_subprocess_env( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify _resolve_agent_runtime_plan forwards host retry settings to Harbor subprocess_env.""" + monkeypatch.setattr( + runner.os, + "environ", + { + "PATH": "/usr/bin", + "SKILL_EVAL_LLM_MAX_RETRIES": "0", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY": "0.1", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY": "0.5", + }, + ) + + plans = runner._resolve_agent_runtime_plan( + provider=_provider("openai"), + agents=["codex"], + models={"codex": "gpt-4o-mini"}, + configured_runtime_env={}, + env_mode="docker", + ) + + subprocess_env = plans["codex"].subprocess_env + assert subprocess_env["SKILL_EVAL_LLM_MAX_RETRIES"] == "0" + assert subprocess_env["SKILL_EVAL_LLM_RETRY_BASE_DELAY"] == "0.1" + assert subprocess_env["SKILL_EVAL_LLM_RETRY_MAX_DELAY"] == "0.5" + assert "SKILL_EVAL_LLM_MAX_RETRIES" not in plans["codex"].staged_env + + @pytest.mark.parametrize("source_name", ["LLM_JUDGE_MODEL", "SKILL_EVAL_JUDGE_MODEL"]) def test_runtime_env_rejects_operator_owned_windows_style_variable_references( monkeypatch: pytest.MonkeyPatch, diff --git a/tests/test_harbor_task_toml_retry_env.py b/tests/test_harbor_task_toml_retry_env.py new file mode 100644 index 000000000..1c8ebb4a4 --- /dev/null +++ b/tests/test_harbor_task_toml_retry_env.py @@ -0,0 +1,388 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Test Harbor task.toml verifier environment forwarding for retry settings.""" + +from __future__ import annotations + +import tomllib +from pathlib import Path +from typing import TYPE_CHECKING, Any + +from harbor.utils.env import resolve_env_vars +from tests.conftest import load_harbor_eval_template + +from skillevaluator.provider_config import ProviderConfig +from skillevaluator.tier3.harbor import runner, runtime_preflight +from skillevaluator.tier3.harbor.adapter import ( + _VERIFIER_PROVIDER_ENV_VARS, + _VERIFIER_RETRY_ENV_VARS, + _verifier_env_block, + _verifier_env_vars, + stage_native_harbor_tasks, +) + +if TYPE_CHECKING: + import pytest +else: + import pytest + + +def test_verifier_provider_env_vars_includes_retry_settings() -> None: + """Verify _VERIFIER_PROVIDER_ENV_VARS allowlist contains all retry configuration keys.""" + expected_retry_vars = { + "SKILL_EVAL_LLM_JUDGE_BUDGET_SEC", + "SKILL_EVAL_LLM_MAX_RETRIES", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY", + } + assert expected_retry_vars == _VERIFIER_RETRY_ENV_VARS + assert expected_retry_vars.issubset(_VERIFIER_PROVIDER_ENV_VARS) + + +def test_verifier_env_block_forwards_retry_settings() -> None: + """Verify _verifier_env_block generates task.toml env lines for staged retry variables.""" + runtime_env = { + "SKILL_EVAL_LLM_JUDGE_BUDGET_SEC": "120.0", + "SKILL_EVAL_LLM_MAX_RETRIES": "5", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY": "2.0", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY": "40.0", + "UNRELATED_HOST_VAR": "secret", + } + env_vars = _verifier_env_vars(runtime_env) + assert "SKILL_EVAL_LLM_JUDGE_BUDGET_SEC" in env_vars + assert "SKILL_EVAL_LLM_MAX_RETRIES" in env_vars + assert "SKILL_EVAL_LLM_RETRY_BASE_DELAY" in env_vars + assert "SKILL_EVAL_LLM_RETRY_MAX_DELAY" in env_vars + assert "UNRELATED_HOST_VAR" not in env_vars + + env_block = _verifier_env_block(runtime_env, indent=" ") + assert ' SKILL_EVAL_LLM_JUDGE_BUDGET_SEC = "${SKILL_EVAL_LLM_JUDGE_BUDGET_SEC}"' in env_block + assert ' SKILL_EVAL_LLM_MAX_RETRIES = "${SKILL_EVAL_LLM_MAX_RETRIES}"' in env_block + assert ' SKILL_EVAL_LLM_RETRY_BASE_DELAY = "${SKILL_EVAL_LLM_RETRY_BASE_DELAY}"' in env_block + assert ' SKILL_EVAL_LLM_RETRY_MAX_DELAY = "${SKILL_EVAL_LLM_RETRY_MAX_DELAY}"' in env_block + assert "UNRELATED_HOST_VAR" not in env_block + + +def _stub_harbor_runner( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + *, + task_source: str, + grading_mode: str = "default", +) -> dict[str, Any]: + """Stub Harbor runner dependencies and capture the staged launch payload.""" + provider = ProviderConfig( + provider="openai", + model="gpt-4o-mini", + api_key="test-openai-key", + base_url="https://api.openai.com/v1", + litellm_model="openai/gpt-4o-mini", + ) + monkeypatch.setattr(runner, "resolve_llm_provider", lambda: provider) + monkeypatch.setattr( + runner, + "load_evals_config", + lambda _path: ( + { + "harbor": {"task_source": task_source, "base_image_mode": "disabled"}, + "grading": {"mode": grading_mode}, + }, + None, + ), + ) + monkeypatch.setattr(runner, "_check_prerequisites", lambda **_kwargs: []) + monkeypatch.setattr( + runtime_preflight, + "probe_model", + lambda selected_provider: runtime_preflight.ModelProbeResult( + True, + selected_provider.provider, + selected_provider.model, + f"model {selected_provider.model} is available", + ), + ) + + captured_launch: dict[str, Any] = {} + + def capture_launch(**kwargs: Any) -> list[str]: + captured_launch["with_skill"] = kwargs["with_skill"] + captured_launch["run_env"] = dict(kwargs["run_env"]) + return [] + + monkeypatch.setattr(runner, "_run_agent_pair", capture_launch) + monkeypatch.setattr( + runner, + "collect_harbor_results", + lambda **_kwargs: {"execution_status": "complete", "execution_errors": [], "metrics": [], "agents": {}}, + ) + + def render_report(_skill_path: Path, output_dir: Path, **_kwargs: Any) -> Path: + report = output_dir / "report.html" + report.write_text("\n", encoding="utf-8") + return report + + monkeypatch.setattr(runner, "render_agent_eval_html_report", render_report) + return captured_launch + + +def _resolve_verifier_retry_tuple( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + verifier_toml_env: dict[str, str], + run_env: dict[str, str], +) -> tuple[int, float, float]: + """Resolve Harbor task.toml [verifier.env] placeholders against run_env and evaluate eval.py retry config.""" + eval_module = load_harbor_eval_template(f"harbor_template_retry_env_{tmp_path.name}") + with monkeypatch.context() as harbor_proc_env: + harbor_proc_env.setattr("os.environ", dict(run_env)) + resolved_verifier_env = resolve_env_vars(verifier_toml_env) + + with monkeypatch.context() as verifier_proc_env: + verifier_proc_env.setattr(eval_module.os, "environ", resolved_verifier_env) + return eval_module._resolve_eval_retry_config() + + +def test_runner_forwards_host_retry_settings_to_staged_task_and_verifier_runtime( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Verify host SKILL_EVAL_LLM_* retry settings flow through runner staging, subprocess env, and eval.py.""" + skill = tmp_path / "demo-skill" + evals = skill / "evals" + evals.mkdir(parents=True) + (skill / "SKILL.md").write_text("---\nname: demo-skill\ndescription: Demo skill.\n---\n# Demo\n", encoding="utf-8") + (evals / "evals.json").write_text( + '[{"id": "case-001", "question": "Verify retry env forwarding.", "expected_answer": "ok", "files": []}]\n', + encoding="utf-8", + ) + + monkeypatch.setenv("SKILL_EVAL_LLM_JUDGE_BUDGET_SEC", "95.0") + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "0") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_BASE_DELAY", "0.1") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "0.5") + + captured_launch = _stub_harbor_runner(monkeypatch, tmp_path, task_source="evals_json") + + result = runner.run_harbor_eval( + skill, + ["codex"], + agent_models={"codex": "gpt-4o-mini"}, + output_dir=tmp_path / "results", + env_mode="docker", + skip_baseline=True, + keep_harbor_jobs=True, + agent_runtime_preflight=False, + ) + + assert "error" not in result + assert "with_skill" in captured_launch + task_toml_path = captured_launch["with_skill"] / "case-001" / "task.toml" + staged_task = tomllib.loads(task_toml_path.read_text(encoding="utf-8")) + verifier_toml_env = staged_task["verifier"]["env"] + + assert verifier_toml_env["SKILL_EVAL_LLM_JUDGE_BUDGET_SEC"] == "${SKILL_EVAL_LLM_JUDGE_BUDGET_SEC}" + assert verifier_toml_env["SKILL_EVAL_LLM_MAX_RETRIES"] == "${SKILL_EVAL_LLM_MAX_RETRIES}" + assert verifier_toml_env["SKILL_EVAL_LLM_RETRY_BASE_DELAY"] == "${SKILL_EVAL_LLM_RETRY_BASE_DELAY}" + assert verifier_toml_env["SKILL_EVAL_LLM_RETRY_MAX_DELAY"] == "${SKILL_EVAL_LLM_RETRY_MAX_DELAY}" + + run_env = captured_launch["run_env"] + assert run_env["SKILL_EVAL_LLM_JUDGE_BUDGET_SEC"] == "95.0" + assert run_env["SKILL_EVAL_LLM_MAX_RETRIES"] == "0" + assert run_env["SKILL_EVAL_LLM_RETRY_BASE_DELAY"] == "0.1" + assert run_env["SKILL_EVAL_LLM_RETRY_MAX_DELAY"] == "0.5" + + assert _resolve_verifier_retry_tuple(monkeypatch, tmp_path, verifier_toml_env, run_env) == (0, 0.1, 0.5) + + +def _create_native_skill_with_authored_retry_env(skill_dir: Path, grading_mode: str) -> None: + """Create a native Harbor skill fixture whose task.toml defines all three retry variables.""" + evals = skill_dir / "evals" + native_task = evals / "harbor" / "case-001" + (native_task / "environment").mkdir(parents=True) + (skill_dir / "SKILL.md").write_text( + "---\nname: demo-skill\ndescription: Demo skill.\n---\n# Demo\n", + encoding="utf-8", + ) + (evals / "evals.json").write_text( + '[{"id": "case-001", "question": "Run native case.", "expected_answer": "ok", "files": []}]\n', + encoding="utf-8", + ) + (evals / "config.yml").write_text( + f"schema_version: 1\nharbor:\n task_source: native_harbor\ngrading:\n mode: {grading_mode}\n", + encoding="utf-8", + ) + if grading_mode == "default_plus_custom": + (evals / "grader.py").write_text("def grade(*args, **kwargs):\n return 1\n", encoding="utf-8") + (native_task / "instruction.md").write_text("Run native case.\n", encoding="utf-8") + (native_task / "environment" / "Dockerfile").write_text("FROM python:3.12-slim\n", encoding="utf-8") + (native_task / "task.toml").write_text( + 'schema_version = "1.3"\n\n' + '[task]\nname = "nvidia/case-001"\n\n' + '[metadata]\nentry_id = "case-001"\n\n' + "[verifier]\ntimeout_sec = 180.0\n\n" + "[verifier.env]\n" + 'SKILL_EVAL_LLM_MAX_RETRIES = "2"\n' + 'SKILL_EVAL_LLM_RETRY_BASE_DELAY = "0.5"\n' + 'SKILL_EVAL_LLM_RETRY_MAX_DELAY = "10.0"\n\n' + "[environment]\n", + encoding="utf-8", + ) + + +@pytest.mark.parametrize("grading_mode", ["default", "default_plus_custom"]) +def test_native_task_preserves_authored_retry_settings_without_host_override( + tmp_path: Path, + grading_mode: str, +) -> None: + """Verify native task staging keeps authored [verifier.env] retry values when host does not override them.""" + skill = tmp_path / f"native-no-override-{grading_mode}" + _create_native_skill_with_authored_retry_env(skill, grading_mode) + + staged = stage_native_harbor_tasks( + skill, + tmp_path / f"staged-no-override-{grading_mode}", + grading_mode=grading_mode, + verifier_env={"SKILL_EVAL_LLM_PROVIDER": "openai", "SKILL_EVAL_LLM_MODEL": "gpt-4o-mini"}, + )[0] + + staged_task = tomllib.loads((staged / "task.toml").read_text(encoding="utf-8")) + verifier_env = staged_task["verifier"]["env"] + assert verifier_env["SKILL_EVAL_LLM_MAX_RETRIES"] == "2" + assert verifier_env["SKILL_EVAL_LLM_RETRY_BASE_DELAY"] == "0.5" + assert verifier_env["SKILL_EVAL_LLM_RETRY_MAX_DELAY"] == "10.0" + assert verifier_env["SKILL_EVAL_LLM_PROVIDER"] == "${SKILL_EVAL_LLM_PROVIDER}" + + +@pytest.mark.parametrize("grading_mode", ["default", "default_plus_custom"]) +@pytest.mark.parametrize( + ("retry_var", "host_val", "expected_tuple"), + [ + ("SKILL_EVAL_LLM_MAX_RETRIES", "0", (0, 0.5, 10.0)), + ("SKILL_EVAL_LLM_RETRY_BASE_DELAY", "0.1", (2, 0.1, 10.0)), + ("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "0.5", (2, 0.5, 0.5)), + ], +) +def test_native_task_host_override_replaces_authored_retry_key_without_duplicate_toml_keys( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + grading_mode: str, + retry_var: str, + host_val: str, + expected_tuple: tuple[int, float, float], +) -> None: + """Verify each host retry override replaces its colliding authored [verifier.env] key and produces valid TOML.""" + skill = tmp_path / f"native-override-{grading_mode}-{retry_var.lower()}" + _create_native_skill_with_authored_retry_env(skill, grading_mode) + + for var in _VERIFIER_RETRY_ENV_VARS: + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv(retry_var, host_val) + + captured_launch = _stub_harbor_runner( + monkeypatch, + tmp_path, + task_source="native_harbor", + grading_mode=grading_mode, + ) + + result = runner.run_harbor_eval( + skill, + ["codex"], + agent_models={"codex": "gpt-4o-mini"}, + output_dir=tmp_path / f"results-{grading_mode}-{retry_var.lower()}", + env_mode="docker", + grading_mode=grading_mode, + skip_baseline=True, + keep_harbor_jobs=True, + agent_runtime_preflight=False, + ) + + assert "error" not in result + task_toml_path = captured_launch["with_skill"] / "case-001" / "task.toml" + staged_task = tomllib.loads(task_toml_path.read_text(encoding="utf-8")) + verifier_toml_env = staged_task["verifier"]["env"] + assert verifier_toml_env[retry_var] == f"${{{retry_var}}}" + + run_env = captured_launch["run_env"] + assert run_env[retry_var] == host_val + + assert _resolve_verifier_retry_tuple(monkeypatch, tmp_path, verifier_toml_env, run_env) == expected_tuple + + +@pytest.mark.parametrize("grading_mode", ["default", "default_plus_custom"]) +def test_native_task_simultaneous_host_overrides_and_quoted_keys( + tmp_path: Path, + grading_mode: str, +) -> None: + """Verify simultaneous host overrides replace all three authored retry keys including quoted TOML keys.""" + skill = tmp_path / f"native-all-three-{grading_mode}" + _create_native_skill_with_authored_retry_env(skill, grading_mode) + task_toml = skill / "evals" / "harbor" / "case-001" / "task.toml" + task_toml.write_text( + task_toml.read_text(encoding="utf-8").replace( + 'SKILL_EVAL_LLM_MAX_RETRIES = "2"\n', + '"SKILL_EVAL_LLM_MAX_RETRIES" = "2"\nCUSTOM_VERIFIER_FLAG = "keep-me"\n', + ), + encoding="utf-8", + ) + + staged = stage_native_harbor_tasks( + skill, + tmp_path / f"staged-all-three-{grading_mode}", + grading_mode=grading_mode, + verifier_env={ + "SKILL_EVAL_LLM_PROVIDER": "openai", + "SKILL_EVAL_LLM_MODEL": "gpt-4o-mini", + "SKILL_EVAL_LLM_MAX_RETRIES": "0", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY": "0.1", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY": "0.5", + }, + )[0] + + staged_task = tomllib.loads((staged / "task.toml").read_text(encoding="utf-8")) + verifier_env = staged_task["verifier"]["env"] + assert verifier_env["SKILL_EVAL_LLM_MAX_RETRIES"] == "${SKILL_EVAL_LLM_MAX_RETRIES}" + assert verifier_env["SKILL_EVAL_LLM_RETRY_BASE_DELAY"] == "${SKILL_EVAL_LLM_RETRY_BASE_DELAY}" + assert verifier_env["SKILL_EVAL_LLM_RETRY_MAX_DELAY"] == "${SKILL_EVAL_LLM_RETRY_MAX_DELAY}" + assert verifier_env["CUSTOM_VERIFIER_FLAG"] == "keep-me" + + +def test_native_task_verifier_env_with_header_comments_and_cross_section_keys( + tmp_path: Path, +) -> None: + """Verify [verifier.env] and next-table headers with inline comments or whitespace do not duplicate tables or keys.""" + skill = tmp_path / "native-header-comments" + _create_native_skill_with_authored_retry_env(skill, "default") + task_toml = skill / "evals" / "harbor" / "case-001" / "task.toml" + task_toml.write_text( + 'schema_version = "1.3"\n\n' + '[task]\nname = "nvidia/case-001"\n\n' + '[metadata]\nentry_id = "case-001"\n\n' + " [verifier] # verifier config\n" + "timeout_sec = 180.0\n\n" + " [verifier.env] # verifier env overrides\n" + 'SKILL_EVAL_LLM_MAX_RETRIES = "2" # authored retry budget\n\n' + '[[steps]] # step list\nname = "step-one"\n\n' + "[steps.verifier.env]\n" + 'SKILL_EVAL_LLM_RETRY_BASE_DELAY = "${SKILL_EVAL_LLM_RETRY_BASE_DELAY}"\n\n' + "[environment] # agent environment\n", + encoding="utf-8", + ) + + staged = stage_native_harbor_tasks( + skill, + tmp_path / "staged-header-comments", + grading_mode="default", + verifier_env={ + "SKILL_EVAL_LLM_PROVIDER": "openai", + "SKILL_EVAL_LLM_MAX_RETRIES": "0", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY": "0.1", + }, + )[0] + + staged_task = tomllib.loads((staged / "task.toml").read_text(encoding="utf-8")) + verifier_env = staged_task["verifier"]["env"] + assert verifier_env["SKILL_EVAL_LLM_MAX_RETRIES"] == "${SKILL_EVAL_LLM_MAX_RETRIES}" + assert verifier_env["SKILL_EVAL_LLM_RETRY_BASE_DELAY"] == "${SKILL_EVAL_LLM_RETRY_BASE_DELAY}" + assert verifier_env["SKILL_EVAL_LLM_PROVIDER"] == "${SKILL_EVAL_LLM_PROVIDER}" diff --git a/tests/test_llm_client_retry.py b/tests/test_llm_client_retry.py new file mode 100644 index 000000000..2e5d26bf5 --- /dev/null +++ b/tests/test_llm_client_retry.py @@ -0,0 +1,206 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Unit tests for LLMClient transparent retry and rate limit handling.""" + +from __future__ import annotations + +from unittest.mock import MagicMock + +import httpx +import pytest + +from skillevaluator.inference.client import LLMClient + + +class MockRateLimitError(Exception): + """Mock rate limit error with HTTP status 429 and headers.""" + + def __init__(self, message: str = "Rate limited", retry_after: str | None = None) -> None: + super().__init__(message) + self.status_code = 429 + headers = {"retry-after": retry_after} if retry_after else {} + self.response = MagicMock(headers=headers) + + +class MockAuthError(Exception): + """Mock auth error with HTTP status 401.""" + + def __init__(self, message: str = "Unauthorized") -> None: + super().__init__(message) + self.status_code = 401 + + +def test_llm_client_retries_on_rate_limit_and_succeeds(monkeypatch: pytest.MonkeyPatch) -> None: + """Verify LLMClient.completions retries on RateLimitError and succeeds on subsequent attempt.""" + slept: list[float] = [] + monkeypatch.setattr("time.sleep", slept.append) + + mock_client = MagicMock() + calls = 0 + + def mock_create(*_args, **_kwargs): + nonlocal calls + calls += 1 + if calls == 1: + raise MockRateLimitError("Rate limit exceeded") + choice = MagicMock() + choice.message.content = "recovered response" + return MagicMock(choices=[choice]) + + mock_client.chat.completions.create = mock_create + + client = LLMClient( + model="gpt-5.6-sol", + base_url="https://api.openai.com/v1", + api_key="test-key", + max_retries=3, + retry_base_delay=0.1, + ) + client._client = mock_client + + result = client.completions("System prompt", "User prompt") + assert result == "recovered response" + assert calls == 2 + assert len(slept) == 1 + + +def test_llm_client_fails_fast_on_auth_error(monkeypatch: pytest.MonkeyPatch) -> None: + """Verify LLMClient.completions raises immediately without retry on AuthenticationError.""" + slept: list[float] = [] + monkeypatch.setattr("time.sleep", slept.append) + + mock_client = MagicMock() + calls = 0 + + def mock_create(*_args, **_kwargs): + nonlocal calls + calls += 1 + raise MockAuthError("Invalid credentials") + + mock_client.chat.completions.create = mock_create + + client = LLMClient( + model="gpt-5.6-sol", + base_url="https://api.openai.com/v1", + api_key="test-key", + max_retries=3, + retry_base_delay=0.1, + ) + client._client = mock_client + + with pytest.raises(MockAuthError): + client.completions("System prompt", "User prompt") + + assert calls == 1 + assert len(slept) == 0 + + +def test_llm_client_exhausts_retries_on_rate_limit(monkeypatch: pytest.MonkeyPatch) -> None: + """Verify LLMClient.completions exhausts retries and raises when 429 persists.""" + slept: list[float] = [] + monkeypatch.setattr("time.sleep", slept.append) + + mock_client = MagicMock() + calls = 0 + + def mock_create(*_args, **_kwargs): + nonlocal calls + calls += 1 + raise MockRateLimitError("Quota exhausted") + + mock_client.chat.completions.create = mock_create + + client = LLMClient( + model="gpt-5.6-sol", + base_url="https://api.openai.com/v1", + api_key="test-key", + max_retries=2, + retry_base_delay=0.1, + ) + client._client = mock_client + + with pytest.raises(MockRateLimitError): + client.completions("System prompt", "User prompt") + + assert calls == 3 # 1 initial + 2 retries + assert len(slept) == 2 + + +def test_llm_client_respects_custom_retry_params() -> None: + """Verify LLMClient stores and exposes custom retry configuration parameters.""" + client = LLMClient( + model="gpt-5.6-sol", + api_key="test-key", + max_retries=5, + retry_base_delay=2.5, + retry_max_delay=45.0, + ) + assert client.max_retries == 5 + assert client.retry_base_delay == 2.5 + assert client.retry_max_delay == 45.0 + + +@pytest.mark.parametrize(("max_retries", "expected_requests"), [(0, 1), (3, 4)]) +def test_openai_transport_level_request_count( + monkeypatch: pytest.MonkeyPatch, + max_retries: int, + expected_requests: int, +) -> None: + """Verify OpenAI SDK executes exactly expected number of requests on rate limits.""" + import openai + + monkeypatch.setattr("time.sleep", lambda _: None) + request_count = 0 + + def mock_handler(request: httpx.Request) -> httpx.Response: + nonlocal request_count + request_count += 1 + return httpx.Response(429, json={"error": {"message": "Rate limit reached", "type": "rate_limit_error"}}) + + client = LLMClient( + model="gpt-5.6-sol", + base_url="https://api.openai.com/v1", + api_key="test-openai-key", + max_retries=max_retries, + retry_base_delay=0.01, + http_client=httpx.Client(transport=httpx.MockTransport(mock_handler)), + ) + with pytest.raises(openai.RateLimitError): + client.completions("system", "user") + + assert request_count == expected_requests + + +@pytest.mark.parametrize(("max_retries", "expected_requests"), [(0, 1), (3, 4)]) +def test_anthropic_transport_level_request_count( + monkeypatch: pytest.MonkeyPatch, + max_retries: int, + expected_requests: int, +) -> None: + """Verify Anthropic SDK executes exactly expected number of requests on rate limits.""" + import anthropic + + monkeypatch.setattr("time.sleep", lambda _: None) + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "anthropic") + monkeypatch.setenv("ANTHROPIC_API_KEY", "test-anthropic-key") + request_count = 0 + + def mock_handler(request: httpx.Request) -> httpx.Response: + nonlocal request_count + request_count += 1 + return httpx.Response( + 429, + json={"type": "error", "error": {"type": "rate_limit_error", "message": "Rate limited"}}, + ) + + client = LLMClient( + model="claude-3-opus", + max_retries=max_retries, + retry_base_delay=0.01, + http_client=httpx.Client(transport=httpx.MockTransport(mock_handler)), + ) + with pytest.raises(anthropic.RateLimitError): + client.completions("system", "user") + + assert request_count == expected_requests diff --git a/tests/test_llm_judge.py b/tests/test_llm_judge.py index 7d9043d86..b523cb173 100644 --- a/tests/test_llm_judge.py +++ b/tests/test_llm_judge.py @@ -5,10 +5,14 @@ from __future__ import annotations +import email.message +import io import json +import urllib.error from unittest.mock import MagicMock, patch import pytest +from tests.conftest import MockUrllibResponse, load_harbor_eval_template from skillevaluator.provider_config import CHAT_CHEAP_OPENAI, CHAT_DEFAULT_OPENAI from skillevaluator.tier3.eval_core import llm_judge @@ -23,6 +27,7 @@ ], ) def test_native_openai_gpt5_uses_max_completion_tokens_without_temperature(model: str) -> None: + """Verify native OpenAI gpt-5 requests use max_completion_tokens without temperature.""" payload = llm_judge._chat_completion_payload( model=model, prompt="Judge this response", @@ -52,10 +57,12 @@ def test_native_openai_gpt5_uses_max_completion_tokens_without_temperature(model ], ) def test_gpt5_family_rejects_custom_temperature(model: str) -> None: + """Verify gpt-5 models do not support custom temperature.""" assert not llm_judge._supports_custom_temperature(model) def test_older_models_accept_custom_temperature() -> None: + """Verify legacy model families support custom temperature.""" assert llm_judge._supports_custom_temperature("gpt-4.1-mini") assert llm_judge._supports_custom_temperature("claude-opus-4-6") assert llm_judge._supports_custom_temperature("claude-opus-4-20250514") @@ -82,12 +89,14 @@ def test_older_models_accept_custom_temperature() -> None: ], ) def test_newer_claude_models_reject_custom_temperature(model: str) -> None: + """Verify newer Claude models do not accept custom temperature.""" assert not llm_judge._supports_custom_temperature(model) def test_call_public_llm_uses_production_gpt5_payload_without_temperature( monkeypatch: pytest.MonkeyPatch, ) -> None: + """Verify call_public_llm formats gpt-5 payload with max_completion_tokens and no temperature.""" monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") monkeypatch.setenv("OPENAI_API_KEY", "test-key") monkeypatch.delenv("SKILL_EVAL_LLM_MODEL", raising=False) @@ -265,3 +274,860 @@ def test_native_openai_completion_token_url_accepts_only_canonical_variants(requ ) def test_deceptive_openai_urls_keep_max_tokens(provider: str, request_url: str) -> None: assert not llm_judge._is_native_openai_chat_url(provider, request_url) + + +@pytest.mark.parametrize( + ("judge_name", "call_args", "expected_schema_name", "valid_response"), + [ + ( + "judge_accuracy", + ("question", "ground truth", "agent response"), + "accuracy_judgment", + json.dumps( + { + "criteria": { + "SKILL_IDENTIFIED": True, + "ACTION_CORRECT": True, + "FACTUALLY_ACCURATE": True, + "TASK_ADDRESSED": True, + "ACTIONABLE": True, + }, + "score": 1.0, + "reason": "all good", + } + ), + ), + ( + "judge_goal_accuracy", + ("question", "ground truth", "agent response"), + "goal_accuracy_judgment", + json.dumps( + { + "user_goal": "do x", + "end_state": "did x", + "achieved": True, + "score": 1.0, + "reason": "achieved", + } + ), + ), + ( + "judge_behavior_check", + ("conversation", ["behavior 1"]), + "behavior_check_judgment", + json.dumps( + { + "results": [{"step": 1, "passed": True, "reason": "observed"}], + "score": 1.0, + "summary": "observed", + } + ), + ), + ], +) +def test_structured_judges_pass_openai_response_format_schema( + monkeypatch: pytest.MonkeyPatch, + judge_name: str, + call_args: tuple, + expected_schema_name: str, + valid_response: str, +) -> None: + """Verify OpenAI/compatible judge calls pass strict json_schema in both shared judge and template.""" + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build") + monkeypatch.setenv("NVIDIA_API_KEY", "test-key") + monkeypatch.delenv("SKILL_EVAL_LLM_MODEL", raising=False) + + mock_openai = MagicMock() + mock_openai.chat.completions.create.return_value = MagicMock( + choices=[MagicMock(message=MagicMock(content=valid_response))] + ) + with patch("openai.OpenAI", return_value=mock_openai): + result = getattr(llm_judge, judge_name)(*call_args) + + assert result["score"] == 1.0 + sdk_kwargs = mock_openai.chat.completions.create.call_args.kwargs + assert "response_format" in sdk_kwargs + rf = sdk_kwargs["response_format"] + assert rf["type"] == "json_schema" + assert rf["json_schema"]["name"] == expected_schema_name + assert rf["json_schema"]["strict"] is True + assert rf["json_schema"]["schema"]["type"] == "object" + assert rf["json_schema"]["schema"]["additionalProperties"] is False + + # Verify Harbor template parity on the HTTP boundary + eval_template = load_harbor_eval_template("harbor_template_eval_slice1") + captured_requests: list[dict] = [] + resp = MockUrllibResponse({"choices": [{"message": {"content": valid_response}}]}) + + monkeypatch.setattr( + eval_template.urllib.request, + "urlopen", + lambda req, **_: captured_requests.append(json.loads(req.data)) or resp, + ) + template_result = getattr(eval_template, judge_name)(*call_args) + assert template_result["score"] == 1.0 + assert len(captured_requests) == 1 + assert captured_requests[0]["response_format"] == rf + + +def test_structured_judges_pass_anthropic_output_config_schema( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify Anthropic judge calls pass output_config.format.json_schema instead of response_format.""" + valid_response = json.dumps( + { + "criteria": { + "SKILL_IDENTIFIED": True, + "ACTION_CORRECT": True, + "FACTUALLY_ACCURATE": True, + "TASK_ADDRESSED": True, + "ACTIONABLE": True, + }, + "score": 1.0, + "reason": "all good", + } + ) + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "anthropic") + monkeypatch.setenv("ANTHROPIC_API_KEY", "test-anthropic-key") + + mock_anthropic = MagicMock() + mock_block = MagicMock() + mock_block.type = "text" + mock_block.text = valid_response + mock_anthropic.messages.create.return_value = MagicMock(content=[mock_block]) + + with patch("anthropic.Anthropic", return_value=mock_anthropic): + result = llm_judge.judge_accuracy("question", "ground truth", "agent response") + + assert result["score"] == 1.0 + anth_kwargs = mock_anthropic.messages.create.call_args.kwargs + assert "response_format" not in anth_kwargs + assert "output_config" in anth_kwargs + assert anth_kwargs["output_config"]["format"]["type"] == "json_schema" + assert anth_kwargs["output_config"]["format"]["schema"] == llm_judge.ACCURACY_JSON_SCHEMA + + eval_template = load_harbor_eval_template("harbor_template_eval_anth") + captured_requests: list[dict] = [] + anth_resp = MockUrllibResponse({"content": [{"type": "text", "text": valid_response}]}) + + monkeypatch.setattr( + eval_template.urllib.request, + "urlopen", + lambda req, **_: captured_requests.append(json.loads(req.data)) or anth_resp, + ) + template_result = eval_template.judge_accuracy("question", "ground truth", "agent response") + assert template_result["score"] == 1.0 + assert "response_format" not in captured_requests[0] + assert captured_requests[0]["output_config"] == anth_kwargs["output_config"] + + +def test_accuracy_and_behavior_prompts_use_boolean_instructions() -> None: + """Verify judge prompts do not instruct 'YES or NO' strings when schema requires booleans.""" + assert "YES or NO" not in llm_judge.ACCURACY_PROMPT + assert "YES (observed) or NO" not in llm_judge.BEHAVIOR_CHECK_PROMPT + + +@pytest.fixture(autouse=True) +def _clear_schema_unsupported_targets(): + from skillevaluator.inference import client as inference_client + + target_set = getattr(inference_client, "_SCHEMA_UNSUPPORTED_TARGETS", None) + if isinstance(target_set, set): + target_set.clear() + yield + if isinstance(target_set, set): + target_set.clear() + + +@pytest.mark.parametrize("status_code", [400, 422]) +def test_client_and_template_downgrade_on_400_422_and_memoize_across_calls( + monkeypatch: pytest.MonkeyPatch, + status_code: int, +) -> None: + """Verify HTTP 400/422 on schema calls downgrades to prompt-only and memoizes the model.""" + valid_response = json.dumps( + { + "criteria": { + "SKILL_IDENTIFIED": True, + "ACTION_CORRECT": True, + "FACTUALLY_ACCURATE": True, + "TASK_ADDRESSED": True, + "ACTIONABLE": True, + }, + "score": 1.0, + "reason": "downgraded ok", + } + ) + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build") + monkeypatch.setenv("NVIDIA_API_KEY", "test-key") + monkeypatch.setenv("SKILL_EVAL_LLM_MODEL", "legacy-unsupported-model") + + class _BadRequestError(Exception): + def __init__(self, code: int): + super().__init__(f"HTTP {code}: response_format of type 'json_schema' is not supported") + self.status_code = code + + sdk_calls: list[dict] = [] + + def fake_sdk_create(**kwargs): + sdk_calls.append(kwargs) + if "response_format" in kwargs: + raise _BadRequestError(status_code) + return MagicMock(choices=[MagicMock(message=MagicMock(content=valid_response))]) + + mock_openai = MagicMock() + mock_openai.chat.completions.create.side_effect = fake_sdk_create + + with patch("openai.OpenAI", return_value=mock_openai): + # Call 1: tries with response_format -> gets 400/422 -> retries without response_format -> succeeds + res1 = llm_judge.judge_accuracy("q1", "gt1", "ans1") + # Call 2: memoized! Should skip response_format on first attempt (only 1 SDK call for Call 2) + res2 = llm_judge.judge_accuracy("q2", "gt2", "ans2") + + assert res1["score"] == 1.0 + assert res2["score"] == 1.0 + assert len(sdk_calls) == 3 + assert "response_format" in sdk_calls[0] + assert "response_format" not in sdk_calls[1] + assert "response_format" not in sdk_calls[2] + + # Verify Harbor template parity for 400/422 downgrade + memoization + eval_template = load_harbor_eval_template(f"harbor_template_eval_{status_code}") + http_requests: list[dict] = [] + ok_resp = MockUrllibResponse({"choices": [{"message": {"content": valid_response}}]}) + + def fake_urlopen(req, **_): + payload = json.loads(req.data) + http_requests.append(payload) + if "response_format" in payload: + raise urllib.error.HTTPError( + req.full_url, + status_code, + "Bad Request", + email.message.Message(), + io.BytesIO(b'{"error": "json_schema not supported"}'), + ) + return ok_resp + + monkeypatch.setattr(eval_template.urllib.request, "urlopen", fake_urlopen) + tres1 = eval_template.judge_accuracy("q1", "gt1", "ans1") + tres2 = eval_template.judge_accuracy("q2", "gt2", "ans2") + + assert tres1["score"] == 1.0 + assert tres2["score"] == 1.0 + assert len(http_requests) == 3 + assert "response_format" in http_requests[0] + assert "response_format" not in http_requests[1] + assert "response_format" not in http_requests[2] + + +def test_transient_429_preserves_schema_and_does_not_memoize_unsupported( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify HTTP 429 retries with backoff while keeping response_format intact.""" + valid_response = json.dumps( + { + "criteria": { + "SKILL_IDENTIFIED": True, + "ACTION_CORRECT": True, + "FACTUALLY_ACCURATE": True, + "TASK_ADDRESSED": True, + "ACTIONABLE": True, + }, + "score": 1.0, + "reason": "retried 429 with schema", + } + ) + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build") + monkeypatch.setenv("NVIDIA_API_KEY", "test-key") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_BASE_DELAY", "0.0") + monkeypatch.setattr("time.sleep", lambda _s: None) + + class _RateLimitError(Exception): + status_code = 429 + + sdk_calls: list[dict] = [] + + def fake_sdk_create(**kwargs): + sdk_calls.append(kwargs) + if len(sdk_calls) == 1: + raise _RateLimitError("rate limited") + return MagicMock(choices=[MagicMock(message=MagicMock(content=valid_response))]) + + mock_openai = MagicMock() + mock_openai.chat.completions.create.side_effect = fake_sdk_create + + with patch("openai.OpenAI", return_value=mock_openai): + res = llm_judge.judge_accuracy("q", "gt", "ans") + + assert res["score"] == 1.0 + assert len(sdk_calls) == 2 + assert "response_format" in sdk_calls[0] + assert "response_format" in sdk_calls[1] + + +def test_template_model_fallback_isolates_schema_memoization_per_model( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify memoizing primary-model as schema-unsupported does not strip schema from fallback-model.""" + import io + import urllib.error + + valid_response = json.dumps( + { + "criteria": { + "SKILL_IDENTIFIED": True, + "ACTION_CORRECT": True, + "FACTUALLY_ACCURATE": True, + "TASK_ADDRESSED": True, + "ACTIONABLE": True, + }, + "score": 1.0, + "reason": "fallback model with schema", + } + ) + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") + monkeypatch.setenv("OPENAI_API_KEY", "test-key") + monkeypatch.setenv("SKILL_EVAL_LLM_BASE_URL", "https://openai-compatible.example/v1") + monkeypatch.setenv("SKILL_EVAL_LLM_MODEL", "old-primary-model") + monkeypatch.setenv("LLM_JUDGE_FALLBACK_MODELS", "modern-fallback-model") + + eval_template = load_harbor_eval_template("harbor_template_eval_fallback_iso") + http_requests: list[dict] = [] + ok_resp = MockUrllibResponse({"choices": [{"message": {"content": valid_response}}]}) + + def fake_urlopen(req, **_): + payload = json.loads(req.data) + http_requests.append(payload) + if payload["model"] == "old-primary-model": + if "response_format" in payload: + raise urllib.error.HTTPError( + req.full_url, + 400, + "Bad Request", + email.message.Message(), + io.BytesIO(b'{"error": "json_schema not supported"}'), + ) + raise urllib.error.HTTPError( + req.full_url, + 404, + "Not Found", + email.message.Message(), + io.BytesIO(b'{"error": "model not found"}'), + ) + return ok_resp + + monkeypatch.setattr(eval_template.urllib.request, "urlopen", fake_urlopen) + res = eval_template.judge_accuracy("q", "gt", "ans") + assert res["score"] == 1.0 + assert [r["model"] for r in http_requests] == [ + "old-primary-model", + "old-primary-model", + "modern-fallback-model", + ] + assert "response_format" in http_requests[0] + assert "response_format" not in http_requests[1] + assert "response_format" in http_requests[2] + + +def test_reasoning_token_exhaustion_missing_message_retries_cleanly( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify missing 'message' or message=None on finish_reason='length' is treated as empty content.""" + valid_response = json.dumps( + { + "criteria": { + "SKILL_IDENTIFIED": True, + "ACTION_CORRECT": True, + "FACTUALLY_ACCURATE": True, + "TASK_ADDRESSED": True, + "ACTIONABLE": True, + }, + "score": 1.0, + "reason": "recovered from missing message", + } + ) + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build") + monkeypatch.setenv("NVIDIA_API_KEY", "test-key") + + mock_openai = MagicMock() + mock_openai.chat.completions.create.side_effect = [ + MagicMock(choices=[MagicMock(message=None)]), + MagicMock(choices=[MagicMock(message=MagicMock(content=valid_response))]), + ] + with patch("openai.OpenAI", return_value=mock_openai): + res = llm_judge.judge_accuracy("q", "gt", "ans") + assert res["score"] == 1.0 + + eval_template = load_harbor_eval_template("harbor_template_eval_missing_msg") + responses = iter( + [ + MockUrllibResponse({"choices": [{"finish_reason": "length", "index": 0, "message": None}]}), + MockUrllibResponse( + {"choices": [{"finish_reason": "stop", "index": 0, "message": {"content": valid_response}}]} + ), + ] + ) + + monkeypatch.setattr(eval_template.urllib.request, "urlopen", lambda _req, **_: next(responses)) + tres = eval_template.judge_accuracy("q", "gt", "ans") + assert tres["score"] == 1.0 + + +def test_persistent_400_after_schema_downgrade_fails_fast_without_infinite_loop( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify if downgraded prompt-only call also returns HTTP 400, it fails fast after 2 calls.""" + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "nv_build") + monkeypatch.setenv("NVIDIA_API_KEY", "test-secret-key-9999999") + monkeypatch.setenv("SKILL_EVAL_LLM_MODEL", "broken-model") + + class _BadRequestError(Exception): + status_code = 400 + + sdk_calls: list[dict] = [] + + def fake_sdk_create(**kwargs): + sdk_calls.append(kwargs) + if "response_format" in kwargs: + raise _BadRequestError("response_format is not supported; echoed test-secret-key-9999999") + raise _BadRequestError("HTTP 400 echoed test-secret-key-9999999") + + mock_openai = MagicMock() + mock_openai.chat.completions.create.side_effect = fake_sdk_create + + with patch("openai.OpenAI", return_value=mock_openai): + res = llm_judge.judge_accuracy("q", "gt", "ans") + + assert res["score"] is None + assert res["status"] == "error" + assert "test-secret-key-9999999" not in res["reason"] + assert "[REDACTED]" in res["reason"] + assert len(sdk_calls) == 2 + assert "response_format" in sdk_calls[0] + assert "response_format" not in sdk_calls[1] + + +@pytest.mark.parametrize( + "schema_error", + ["output_config: Extra inputs are not permitted", "output_config.format is not supported"], +) +def test_anthropic_downgrades_on_400_and_memoizes_across_calls( + monkeypatch: pytest.MonkeyPatch, schema_error: str +) -> None: + """Verify Anthropic provider downgrades output_config on HTTP 400 and memoizes across calls.""" + import io + import urllib.error + + valid_response = json.dumps( + { + "criteria": { + "SKILL_IDENTIFIED": True, + "ACTION_CORRECT": True, + "FACTUALLY_ACCURATE": True, + "TASK_ADDRESSED": True, + "ACTIONABLE": True, + }, + "score": 1.0, + "reason": "anthropic downgraded ok", + } + ) + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "anthropic") + monkeypatch.setenv("ANTHROPIC_API_KEY", "test-anthropic-key") + monkeypatch.setenv("SKILL_EVAL_LLM_MODEL", "claude-3-haiku-20240307") + + class _BadRequestError(Exception): + status_code = 400 + + anth_calls: list[dict] = [] + + def fake_messages_create(**kwargs): + anth_calls.append(kwargs) + if "output_config" in kwargs: + raise _BadRequestError(schema_error) + block = MagicMock() + block.type = "text" + block.text = valid_response + return MagicMock(content=[block]) + + mock_anthropic = MagicMock() + mock_anthropic.messages.create.side_effect = fake_messages_create + + with patch("anthropic.Anthropic", return_value=mock_anthropic): + res1 = llm_judge.judge_accuracy("q1", "gt1", "ans1") + res2 = llm_judge.judge_accuracy("q2", "gt2", "ans2") + + assert res1["score"] == 1.0 + assert res2["score"] == 1.0 + assert len(anth_calls) == 3 + assert "output_config" in anth_calls[0] + assert "output_config" not in anth_calls[1] + assert "output_config" not in anth_calls[2] + + eval_template = load_harbor_eval_template("harbor_template_eval_anth_400") + http_requests: list[dict] = [] + anth_resp = MockUrllibResponse({"content": [{"type": "text", "text": valid_response}]}) + + def fake_urlopen(req, timeout=90): + payload = json.loads(req.data) + http_requests.append(payload) + if "output_config" in payload: + raise urllib.error.HTTPError( + req.full_url, + 400, + "Bad Request", + email.message.Message(), + io.BytesIO(json.dumps({"error": schema_error}).encode()), + ) + return anth_resp + + monkeypatch.setattr(eval_template.urllib.request, "urlopen", fake_urlopen) + tres1 = eval_template.judge_accuracy("q1", "gt1", "ans1") + tres2 = eval_template.judge_accuracy("q2", "gt2", "ans2") + + assert tres1["score"] == 1.0 + assert tres2["score"] == 1.0 + assert len(http_requests) == 3 + assert "output_config" in http_requests[0] + assert "output_config" not in http_requests[1] + assert "output_config" not in http_requests[2] + + +def test_judge_schema_not_spoofed_by_transcript_containing_accuracy_tokens(monkeypatch): + """Ensure judge_goal_accuracy and judge_behavior_check pass explicit schemas and are not spoofed by SKILL_IDENTIFIED/ACTION_CORRECT in the agent transcript.""" + import sys + from types import SimpleNamespace + + from skillevaluator.inference import client as client_mod + from skillevaluator.tier3.eval_core import llm_judge + + client_mod._SCHEMA_UNSUPPORTED_TARGETS.clear() + monkeypatch.setenv("LLM_JUDGE_PROVIDER", "openai") + monkeypatch.setenv("LLM_JUDGE_MODEL", "google/gemini-3.8-flash") + monkeypatch.setenv("OPENAI_API_KEY", "test-openai-key") + + captured_kwargs: list[dict] = [] + + class FakeCompletions: + def create(self, **kwargs): + captured_kwargs.append(kwargs) + schema_name = kwargs.get("response_format", {}).get("json_schema", {}).get("name") + if schema_name == "goal_accuracy_judgment": + content = '{"user_goal": "g", "end_state": "e", "achieved": true, "score": 1.0, "reason": "ok"}' + else: + content = '{"results": [{"step": 1, "passed": true, "reason": "ok"}], "score": 1.0, "summary": "ok"}' + return SimpleNamespace(choices=[SimpleNamespace(message=SimpleNamespace(content=content))]) + + class FakeOpenAI: + def __init__(self, **_kwargs): + self.chat = SimpleNamespace(completions=FakeCompletions()) + + monkeypatch.setitem(sys.modules, "openai", SimpleNamespace(OpenAI=FakeOpenAI)) + + adversarial_transcript = "Checked SKILL_IDENTIFIED and ACTION_CORRECT in llm_judge.py" + goal_res = llm_judge.judge_goal_accuracy("goal", "expected", adversarial_transcript) + beh_res = llm_judge.judge_behavior_check(adversarial_transcript, ["rule"]) + + assert goal_res["score"] == 1.0 + assert beh_res["score"] == 1.0 + assert captured_kwargs[0]["response_format"]["json_schema"]["name"] == "goal_accuracy_judgment" + assert captured_kwargs[1]["response_format"]["json_schema"]["name"] == "behavior_check_judgment" + + +def test_schema_builders_and_downgrade_warning_log(monkeypatch, caplog): + """Verify _build_openai_response_format, _build_anthropic_output_config, and warning log on 400 downgrade.""" + import logging + import sys + from types import SimpleNamespace + + from skillevaluator.inference import client as client_mod + from skillevaluator.tier3.eval_core import llm_judge + + assert client_mod._build_openai_response_format({"type": "object"}, "my_schema") == { + "type": "json_schema", + "json_schema": { + "name": "my_schema", + "strict": True, + "schema": {"type": "object"}, + }, + } + assert client_mod._build_anthropic_output_config({"type": "object"}) == { + "format": { + "type": "json_schema", + "schema": {"type": "object"}, + } + } + + client_mod._SCHEMA_UNSUPPORTED_TARGETS.clear() + monkeypatch.setenv("LLM_JUDGE_PROVIDER", "openai") + monkeypatch.setenv("LLM_JUDGE_MODEL", "legacy/model-400") + monkeypatch.setenv("OPENAI_API_KEY", "test-openai-key") + + valid_response = json.dumps( + { + "criteria": { + "SKILL_IDENTIFIED": True, + "ACTION_CORRECT": True, + "FACTUALLY_ACCURATE": True, + "TASK_ADDRESSED": True, + "ACTIONABLE": True, + }, + "score": 1.0, + "reason": "ok", + } + ) + + class FakeBadRequestError(Exception): + def __init__(self, message: str): + super().__init__(message) + self.status_code = 400 + + class FakeCompletions: + def create(self, **kwargs): + if "response_format" in kwargs: + raise FakeBadRequestError("400 response_format not supported") + return SimpleNamespace(choices=[SimpleNamespace(message=SimpleNamespace(content=valid_response))]) + + class FakeOpenAI: + def __init__(self, **_kwargs): + self.chat = SimpleNamespace(completions=FakeCompletions()) + + monkeypatch.setitem(sys.modules, "openai", SimpleNamespace(OpenAI=FakeOpenAI)) + + with caplog.at_level(logging.WARNING): + res = llm_judge.judge_accuracy("q", "gt", "ans") + + assert res["score"] == 1.0 + assert any("Structured output schema unsupported" in rec.message for rec in caplog.records) + + +@pytest.mark.parametrize( + "error_message", + [ + "context_length_exceeded: maximum context length is 8192 tokens", + "Invalid parameter: max_tokens. response_format is supported for this model.", + "response_format is supported but max_tokens is not allowed", + ], +) +def test_unrelated_400_does_not_disable_schema_flags(error_message: str) -> None: + """An unrelated bad parameter must not disable schema even when the error mentions it.""" + import httpx + import openai + + from skillevaluator.inference import client as client_mod + from skillevaluator.inference.client import LLMClient + + client_mod._SCHEMA_UNSUPPORTED_TARGETS.clear() + schema_flags_sent: list[bool] = [] + + def mock_handler(request: httpx.Request) -> httpx.Response: + payload = json.loads(request.read()) + schema_flags_sent.append("response_format" in payload) + if len(schema_flags_sent) <= 2: + return httpx.Response( + 400, + json={ + "error": { + "message": error_message, + "type": "invalid_request_error", + "code": "context_length_exceeded", + } + }, + ) + return httpx.Response( + 200, + json={ + "choices": [ + { + "message": { + "content": json.dumps( + { + "criteria": { + "SKILL_IDENTIFIED": True, + "ACTION_CORRECT": True, + "FACTUALLY_ACCURATE": True, + "TASK_ADDRESSED": True, + "ACTIONABLE": True, + }, + "score": 1.0, + "reason": "ok", + } + ) + } + } + ] + }, + ) + + client = LLMClient( + model="gpt-5.6-sol", + base_url="https://api.openai.com/v1", + api_key="test-openai-key", + max_retries=0, + http_client=httpx.Client(transport=httpx.MockTransport(mock_handler)), + ) + + with pytest.raises(openai.BadRequestError): + client.completions("sys", "oversized_1", response_schema={"type": "object"}) + + with pytest.raises(openai.BadRequestError): + client.completions("sys", "oversized_2", response_schema={"type": "object"}) + + res = client.completions("sys", "valid_3", response_schema={"type": "object"}) + assert res + + assert schema_flags_sent == [True, True, True] + assert len(client_mod._SCHEMA_UNSUPPORTED_TARGETS) == 0 + + +def test_schema_downgrade_memoized_only_on_confirmed_success() -> None: + """Verify target is only memoized into _SCHEMA_UNSUPPORTED_TARGETS when fallback succeeds.""" + import httpx + import openai + + from skillevaluator.inference import client as client_mod + from skillevaluator.inference.client import LLMClient + + client_mod._SCHEMA_UNSUPPORTED_TARGETS.clear() + target_key = ("openai", "https://api.openai.com/v1", "gpt-5.6-sol") + + calls_a: list[bool] = [] + + def mock_handler_failure(request: httpx.Request) -> httpx.Response: + payload = json.loads(request.read()) + calls_a.append("response_format" in payload) + if "response_format" in payload: + return httpx.Response(400, json={"error": {"message": "response_format is not supported"}}) + return httpx.Response(500, json={"error": {"message": "Internal server error"}}) + + client_a = LLMClient( + model="gpt-5.6-sol", + base_url="https://api.openai.com/v1", + api_key="test-openai-key", + max_retries=0, + http_client=httpx.Client(transport=httpx.MockTransport(mock_handler_failure)), + ) + + with pytest.raises(openai.InternalServerError): + client_a.completions("sys", "prompt", response_schema={"type": "object"}) + + assert calls_a == [True, False] + assert target_key not in client_mod._SCHEMA_UNSUPPORTED_TARGETS + + calls_b: list[bool] = [] + + def mock_handler_success(request: httpx.Request) -> httpx.Response: + payload = json.loads(request.read()) + calls_b.append("response_format" in payload) + if "response_format" in payload: + return httpx.Response(400, json={"error": {"message": "response_format is not supported"}}) + return httpx.Response( + 200, + json={"choices": [{"message": {"content": "ok"}}]}, + ) + + client_b = LLMClient( + model="gpt-5.6-sol", + base_url="https://api.openai.com/v1", + api_key="test-openai-key", + max_retries=0, + http_client=httpx.Client(transport=httpx.MockTransport(mock_handler_success)), + ) + + res = client_b.completions("sys", "prompt", response_schema={"type": "object"}) + assert res == "ok" + assert calls_b == [True, False] + assert target_key in client_mod._SCHEMA_UNSUPPORTED_TARGETS + + +@pytest.mark.parametrize( + "error_message", + [ + "context_length_exceeded", + "Invalid parameter: max_tokens. response_format is supported for this model.", + "response_format is supported but max_tokens is not allowed", + ], +) +def test_harbor_eval_template_unrelated_400_does_not_disable_schema( + monkeypatch: pytest.MonkeyPatch, error_message: str +) -> None: + """Verify template eval.py does not treat unrelated 400 as schema capability failure.""" + eval_template = load_harbor_eval_template("harbor_template_eval_unrelated_400") + + eval_template._SCHEMA_UNSUPPORTED_TARGETS.clear() + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") + monkeypatch.setenv("OPENAI_API_KEY", "test-key-fake") + + http_requests: list[dict] = [] + ok_resp = MockUrllibResponse( + { + "choices": [ + { + "message": { + "content": json.dumps( + { + "criteria": { + "SKILL_IDENTIFIED": True, + "ACTION_CORRECT": True, + "FACTUALLY_ACCURATE": True, + "TASK_ADDRESSED": True, + "ACTIONABLE": True, + }, + "score": 1.0, + "reason": "ok", + } + ) + } + } + ] + } + ) + + call_count = 0 + + def fake_urlopen(req, timeout=90): + nonlocal call_count + call_count += 1 + payload = json.loads(req.data) + http_requests.append(payload) + if call_count == 1: + raise urllib.error.HTTPError( + req.full_url, + 400, + "Bad Request", + email.message.Message(), + io.BytesIO(json.dumps({"error": {"message": error_message}}).encode()), + ) + return ok_resp + + monkeypatch.setattr(eval_template.urllib.request, "urlopen", fake_urlopen) + + res1 = eval_template.judge_accuracy("q1", "gt1", "ans1") + assert res1["status"] == "error" + assert res1["score"] is None + assert len(http_requests) == 1 + assert "response_format" in http_requests[0] + assert len(eval_template._SCHEMA_UNSUPPORTED_TARGETS) == 0 + + res2 = eval_template.judge_accuracy("q2", "gt2", "ans2") + assert res2["score"] == 1.0 + assert len(http_requests) == 2 + assert "response_format" in http_requests[1] + assert len(eval_template._SCHEMA_UNSUPPORTED_TARGETS) == 0 + + +@pytest.mark.parametrize( + "message", + [ + "Unrecognized request argument supplied: response_format", + "response_format is an unknown parameter", + "output_config.format is an unexpected argument", + ], +) +def test_schema_rejection_grammar_accepts_direct_option_errors(message: str) -> None: + """Verify structured schema rejection regex matches direct option error messages.""" + from skillevaluator.inference import client as client_mod + + eval_template = load_harbor_eval_template("harbor_template_eval_rejection_grammar") + assert client_mod._message_rejects_schema_option(message) + assert eval_template._message_rejects_schema_option(message) diff --git a/tests/test_retry.py b/tests/test_retry.py new file mode 100644 index 000000000..2634eeff9 --- /dev/null +++ b/tests/test_retry.py @@ -0,0 +1,694 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Unit tests for zero-dependency retry and rate limit utilities.""" + +from __future__ import annotations + +import urllib.error +from collections.abc import Callable +from datetime import UTC, datetime, timedelta +from email.message import Message +from email.utils import format_datetime +from typing import Any + +import pytest +from tests.conftest import load_harbor_eval_template + +from skillevaluator.inference.retry import ( + DEFAULT_BASE_DELAY, + DEFAULT_MAX_DELAY, + DEFAULT_MAX_RETRIES, + calculate_full_jitter_delay, + is_retriable_status_code, + parse_retry_after, + resolve_retry_config, + retry_call_with_backoff, +) + + +def _http_headers(retry_after: str | None = None) -> Message: + """Build an HTTP header Message for urllib.error.HTTPError tests.""" + headers = Message() + if retry_after is not None: + headers["Retry-After"] = retry_after + return headers + + +@pytest.mark.parametrize( + ("code", "expected"), + [ + (429, True), + (500, True), + (502, True), + (503, True), + (504, True), + (400, False), + (401, False), + (403, False), + (404, False), + (200, False), + (None, False), + ], +) +def test_is_retriable_status_code(code: int | None, expected: bool) -> None: + """Verify is_retriable_status_code correctly classifies transient status codes.""" + assert is_retriable_status_code(code) is expected + + +@pytest.mark.parametrize( + ("header_val", "fallback", "expected"), + [ + ("10", 1.0, 10.0), + ("0", 1.0, 0.0), + ("2.5", 1.0, 2.5), + ("-5", 1.0, 0.0), + ("", 3.0, 3.0), + (None, 4.0, 4.0), + ("invalid", 2.0, 2.0), + ], +) +def test_parse_retry_after_numeric_and_fallback( + header_val: str | None, + fallback: float, + expected: float, +) -> None: + """Verify parse_retry_after handles numeric strings, negatives, and fallbacks.""" + assert parse_retry_after(header_val, fallback) == pytest.approx(expected) + + +def test_parse_retry_after_http_date_future() -> None: + """Verify parse_retry_after handles RFC-7231 HTTP dates in UTC.""" + future_time = datetime.now(UTC) + timedelta(seconds=15) + formatted = format_datetime(future_time, usegmt=True) + + result = parse_retry_after(formatted, fallback_delay=1.0) + assert 13.0 <= result <= 16.0 + + +def test_parse_retry_after_http_date_past() -> None: + """Verify parse_retry_after clamps past dates to 0.0.""" + past_time = datetime.now(UTC) - timedelta(seconds=15) + formatted = format_datetime(past_time, usegmt=True) + + result = parse_retry_after(formatted, fallback_delay=1.0) + assert result == 0.0 + + +def test_parse_retry_after_naive_date() -> None: + """Verify parse_retry_after correctly handles offset-naive parsed datetimes without raising.""" + result = parse_retry_after("Sun, 06 Nov 1994 08:49:37", fallback_delay=5.0) + assert result == 0.0 + + +@pytest.mark.parametrize("attempt", [0, 1, 2, 3, 5, 10]) +def test_calculate_full_jitter_delay_bounds(attempt: int) -> None: + """Verify calculate_full_jitter_delay generates non-negative values bounded by backoff.""" + base_delay = 1.0 + max_delay = 30.0 + expected_ceiling = min(max_delay, base_delay * (2.0**attempt)) + + for _ in range(25): + delay = calculate_full_jitter_delay(attempt, base_delay=base_delay, max_delay=max_delay) + assert 0.0 <= delay <= expected_ceiling + + +def test_resolve_retry_config_defaults() -> None: + """Verify resolve_retry_config returns sensible defaults when environment is empty.""" + config = resolve_retry_config({}) + assert config.max_retries == DEFAULT_MAX_RETRIES + assert config.base_delay == DEFAULT_BASE_DELAY + assert config.max_delay == DEFAULT_MAX_DELAY + + +def test_resolve_retry_config_custom_env() -> None: + """Verify resolve_retry_config accepts standard SKILL_EVAL_LLM_* environment variables.""" + env = { + "SKILL_EVAL_LLM_MAX_RETRIES": "5", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY": "2.5", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY": "45.0", + } + config = resolve_retry_config(env) + assert config.max_retries == 5 + assert config.base_delay == 2.5 + assert config.max_delay == 45.0 + + +def test_resolve_retry_config_defensive_fallbacks() -> None: + """Verify resolve_retry_config safely falls back to defaults when values are malformed.""" + env = { + "SKILL_EVAL_LLM_MAX_RETRIES": "invalid", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY": "-1.0", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY": "not-a-float", + } + config = resolve_retry_config(env) + assert config.max_retries == DEFAULT_MAX_RETRIES + assert config.base_delay == DEFAULT_BASE_DELAY + assert config.max_delay == DEFAULT_MAX_DELAY + + +def test_resolve_retry_config_negative_values_fallback() -> None: + """Verify resolve_retry_config falls back to defaults when values are negative.""" + env = { + "SKILL_EVAL_LLM_MAX_RETRIES": "-5", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY": "-2.0", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY": "-10.0", + } + config = resolve_retry_config(env) + assert config.max_retries == DEFAULT_MAX_RETRIES + assert config.base_delay == DEFAULT_BASE_DELAY + assert config.max_delay == DEFAULT_MAX_DELAY + + +@pytest.mark.parametrize("variable", ["SKILL_EVAL_LLM_RETRY_BASE_DELAY", "SKILL_EVAL_LLM_RETRY_MAX_DELAY"]) +def test_nonfinite_retry_delay_falls_back_in_host_and_verifier(monkeypatch: pytest.MonkeyPatch, variable: str) -> None: + """Verify non-finite retry delay configuration falls back to defaults in host and verifier.""" + config = resolve_retry_config({variable: "inf"}) + assert config.base_delay == DEFAULT_BASE_DELAY + assert config.max_delay == DEFAULT_MAX_DELAY + + verifier = load_harbor_eval_template("harbor_template_retry_nonfinite") + monkeypatch.setenv(variable, "inf") + assert verifier._resolve_eval_retry_config() == ( + verifier._DEFAULT_MAX_RETRIES, + verifier._DEFAULT_BASE_DELAY, + verifier._DEFAULT_MAX_DELAY, + ) + + +def test_verifier_preserves_rate_limit_error_body_when_retry_after_exceeds_cap( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify the terminal 429 report retains its provider diagnostic body when Retry-After exceeds max_delay.""" + import io + + verifier = load_harbor_eval_template("harbor_template_retry_after_diagnostic") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "30") + + def rate_limited(request: Any, timeout: float = 90) -> Any: + _ = timeout + raise urllib.error.HTTPError( + request.full_url, + 429, + "Too Many Requests", + _http_headers("3600"), + io.BytesIO(b'{"error":"capacity returns in one hour"}'), + ) + + monkeypatch.setattr(verifier.urllib.request, "urlopen", rate_limited) + request = verifier.urllib.request.Request("https://example.test/v1/chat/completions") + with pytest.raises(urllib.error.HTTPError) as exc_info: + verifier._urlopen_with_retry(request) + + assert "capacity returns in one hour" in verifier._format_http_error(exc_info.value) + + +def test_retry_call_with_backoff_immediate_success() -> None: + """Verify retry_call_with_backoff executes operation and returns immediately on success.""" + calls = 0 + + def op() -> str: + nonlocal calls + calls += 1 + return "success" + + result = retry_call_with_backoff(op) + assert result == "success" + assert calls == 1 + + +def test_retry_call_with_backoff_succeeds_after_transient_failure(monkeypatch: pytest.MonkeyPatch) -> None: + """Verify retry_call_with_backoff retries transient 429 and succeeds on subsequent attempt.""" + calls = 0 + slept: list[float] = [] + monkeypatch.setattr("time.sleep", slept.append) + + def op() -> str: + nonlocal calls + calls += 1 + if calls == 1: + raise urllib.error.HTTPError("http://api", 429, "Too Many Requests", hdrs=_http_headers(), fp=None) + return "recovered" + + result = retry_call_with_backoff(op, max_retries=3, base_delay=1.0) + assert result == "recovered" + assert calls == 2 + assert len(slept) == 1 + assert slept[0] >= 0.0 + + +def test_retry_call_with_backoff_respects_retry_after_header(monkeypatch: pytest.MonkeyPatch) -> None: + """Verify retry_call_with_backoff uses Retry-After header when present on HTTPError.""" + calls = 0 + slept: list[float] = [] + monkeypatch.setattr("time.sleep", slept.append) + + def op() -> str: + nonlocal calls + calls += 1 + if calls == 1: + raise urllib.error.HTTPError( + "http://api", + 429, + "Too Many Requests", + hdrs=_http_headers("3.5"), + fp=None, + ) + return "done" + + result = retry_call_with_backoff(op, max_retries=2, base_delay=0.1) + assert result == "done" + assert calls == 2 + assert len(slept) == 1 + # 3.5 + small jitter buffer (+0.1-0.5) + assert 3.5 <= slept[0] <= 4.1 + + +def test_retry_call_with_backoff_fails_fast_on_non_retriable_error() -> None: + """Verify retry_call_with_backoff raises immediately without retrying on non-retriable error.""" + calls = 0 + + def op() -> str: + nonlocal calls + calls += 1 + raise urllib.error.HTTPError("http://api", 401, "Unauthorized", hdrs=_http_headers(), fp=None) + + with pytest.raises(urllib.error.HTTPError) as exc_info: + retry_call_with_backoff(op, max_retries=3) + + assert exc_info.value.code == 401 + assert calls == 1 + + +def test_retry_call_with_backoff_exhausts_retries(monkeypatch: pytest.MonkeyPatch) -> None: + """Verify retry_call_with_backoff exhausts retries and raises on persistent 429.""" + calls = 0 + slept: list[float] = [] + monkeypatch.setattr("time.sleep", slept.append) + + def op() -> str: + nonlocal calls + calls += 1 + raise urllib.error.HTTPError("http://api", 429, "Too Many Requests", hdrs=_http_headers(), fp=None) + + with pytest.raises(urllib.error.HTTPError) as exc_info: + retry_call_with_backoff(op, max_retries=3) + + assert exc_info.value.code == 429 + assert calls == 4 # 1 initial + 3 retries + assert len(slept) == 3 + + +def test_retry_call_with_backoff_caps_delay_at_max_delay(monkeypatch: pytest.MonkeyPatch) -> None: + """Verify retry_call_with_backoff caps single attempt sleep duration to max_delay.""" + calls = 0 + slept: list[float] = [] + monkeypatch.setattr("time.sleep", slept.append) + + def op() -> str: + nonlocal calls + calls += 1 + if calls <= 2: + raise urllib.error.HTTPError("http://api", 503, "Service Unavailable", hdrs=_http_headers(), fp=None) + return "finished" + + # With base_delay=100.0 and max_delay=5.0, delay is capped to 5.0 + result = retry_call_with_backoff(op, max_retries=3, base_delay=100.0, max_delay=5.0) + assert result == "finished" + assert calls == 3 + assert len(slept) == 2 + for duration in slept: + assert duration <= 5.0 + + +def test_retry_call_with_backoff_raises_if_retry_after_exceeds_max_delay(monkeypatch: pytest.MonkeyPatch) -> None: + """Verify retry_call_with_backoff raises immediately without sleeping if Retry-After exceeds max_delay.""" + calls = 0 + slept: list[float] = [] + monkeypatch.setattr("time.sleep", slept.append) + + def op() -> str: + nonlocal calls + calls += 1 + raise urllib.error.HTTPError( + "http://api", + 429, + "Too Many Requests", + hdrs=_http_headers("3600"), + fp=None, + ) + + with pytest.raises(urllib.error.HTTPError) as exc_info: + retry_call_with_backoff(op, max_retries=3, max_delay=30.0) + + assert exc_info.value.code == 429 + assert calls == 1 + assert len(slept) == 0 + + +def test_retry_call_with_backoff_invokes_on_retry_callback(monkeypatch: pytest.MonkeyPatch) -> None: + """Verify retry_call_with_backoff invokes on_retry callback with attempt and sleep duration.""" + calls = 0 + callback_calls: list[tuple[int, float]] = [] + monkeypatch.setattr("time.sleep", lambda _: None) + + def on_retry_cb(exc: BaseException, attempt: int, delay: float) -> None: + _ = exc + callback_calls.append((attempt, delay)) + + def op() -> str: + nonlocal calls + calls += 1 + if calls == 1: + raise urllib.error.HTTPError("http://api", 503, "Service Unavailable", hdrs=_http_headers(), fp=None) + return "ok" + + result = retry_call_with_backoff(op, max_retries=2, base_delay=0.1, on_retry=on_retry_cb) + assert result == "ok" + assert calls == 2 + assert len(callback_calls) == 1 + assert callback_calls[0][0] == 1 + assert callback_calls[0][1] >= 0.0 + + +class _BedrockTransportHarness: + """Test harness for driving _call_bedrock against botocore's URLLib3Session.send.""" + + def __init__(self, verifier: Any, monkeypatch: pytest.MonkeyPatch) -> None: + import botocore.httpsession + + self.verifier = verifier + self.monkeypatch = monkeypatch + self.slept: list[float] = [] + self.requests_sent = 0 + self._handler: Callable[[Any], Any] | None = None + monkeypatch.setattr(verifier.time, "sleep", self.slept.append) + monkeypatch.setenv("AWS_ACCESS_KEY_ID", "test-access-key") + monkeypatch.setenv("AWS_SECRET_ACCESS_KEY", "test-secret-key") + monkeypatch.setenv("AWS_REGION", "us-west-2") + + def _send(_session: Any, request: Any) -> Any: + self.requests_sent += 1 + assert self._handler is not None, "Configure harness handler before calling call_bedrock()" + return self._handler(request) + + monkeypatch.setattr(botocore.httpsession.URLLib3Session, "send", _send) + + def set_handler(self, handler: Callable[[Any], Any]) -> None: + """Register a per-request transport handler.""" + self._handler = handler + + @staticmethod + def json_response( + request: Any, + status_code: int, + payload: dict[str, Any], + headers: dict[str, str] | None = None, + ) -> Any: + """Build a botocore AWSResponse with a JSON body.""" + import json + + import botocore.awsrequest + + merged_headers = {"Content-Type": "application/json", **(headers or {})} + resp = botocore.awsrequest.AWSResponse(request.url, status_code, merged_headers, None) + resp._content = json.dumps(payload).encode("utf-8") + return resp + + def throttled_response(self, request: Any, *, retry_after: str | None = None) -> Any: + """Build an HTTP 429 ThrottlingException response.""" + headers = {"x-amzn-errortype": "ThrottlingException"} + if retry_after is not None: + headers["Retry-After"] = retry_after + return self.json_response(request, 429, {"message": "Rate exceeded"}, headers) + + def ok_response(self, request: Any, text: str = "ok") -> Any: + """Build an HTTP 200 Converse output response.""" + return self.json_response( + request, + 200, + {"output": {"message": {"content": [{"text": text}]}}}, + ) + + def call_bedrock(self, prompt: str = "Judge this") -> tuple[str | None, str | None]: + """Invoke verifier._call_bedrock with default test model parameters.""" + return self.verifier._call_bedrock( + prompt, + "us.anthropic.claude-3-5-sonnet-20241022-v2:0", + 512, + 0.0, + ) + + +@pytest.fixture +def bedrock_harness(monkeypatch: pytest.MonkeyPatch) -> _BedrockTransportHarness: + """Provide an isolated verifier template module wired to a botocore transport harness.""" + verifier = load_harbor_eval_template("harbor_template_bedrock_harness") + return _BedrockTransportHarness(verifier, monkeypatch) + + +def test_bedrock_converse_transport_zero_retries_overrides_aws_max_attempts( + bedrock_harness: _BedrockTransportHarness, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify _call_bedrock with SKILL_EVAL_LLM_MAX_RETRIES=0 sends only 1 request even when AWS_MAX_ATTEMPTS=4.""" + monkeypatch.setenv("AWS_MAX_ATTEMPTS", "4") + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "0") + bedrock_harness.set_handler( + lambda req: ( + bedrock_harness.throttled_response(req) + if bedrock_harness.requests_sent == 1 + else bedrock_harness.ok_response(req, "recovered") + ) + ) + + content, error = bedrock_harness.call_bedrock() + assert content is None + assert error is not None and "Bedrock request failed" in error + assert bedrock_harness.requests_sent == 1 + assert bedrock_harness.slept == [] + + +def test_bedrock_converse_transport_positive_retries_overrides_aws_max_attempts_one( + bedrock_harness: _BedrockTransportHarness, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify _call_bedrock with SKILL_EVAL_LLM_MAX_RETRIES=3 retries 429 and succeeds even when AWS_MAX_ATTEMPTS=1.""" + monkeypatch.setenv("AWS_MAX_ATTEMPTS", "1") + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "3") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_BASE_DELAY", "0.05") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "0.2") + bedrock_harness.set_handler( + lambda req: ( + bedrock_harness.throttled_response(req) + if bedrock_harness.requests_sent <= 2 + else bedrock_harness.ok_response(req, "recovered verdict") + ) + ) + + content, error = bedrock_harness.call_bedrock() + assert (content, error) == ("recovered verdict", None) + assert bedrock_harness.requests_sent == 3 + assert len(bedrock_harness.slept) == 2 + assert all(0.0 <= d <= 0.2 for d in bedrock_harness.slept) + + +def test_bedrock_converse_transport_exhausts_configured_retries_on_persistent_429( + bedrock_harness: _BedrockTransportHarness, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify _call_bedrock with SKILL_EVAL_LLM_MAX_RETRIES=3 sends 4 total requests on persistent 429.""" + monkeypatch.setenv("AWS_MAX_ATTEMPTS", "1") + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "3") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_BASE_DELAY", "0.01") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "0.1") + bedrock_harness.set_handler(bedrock_harness.throttled_response) + + content, error = bedrock_harness.call_bedrock() + assert content is None + assert error is not None and "Bedrock request failed" in error + assert bedrock_harness.requests_sent == 4 + assert len(bedrock_harness.slept) == 3 + + +def test_bedrock_converse_transport_fails_fast_on_non_retriable_400( + bedrock_harness: _BedrockTransportHarness, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify _call_bedrock does not retry non-retriable HTTP 400 ValidationException.""" + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "3") + bedrock_harness.set_handler( + lambda req: bedrock_harness.json_response( + req, + 400, + {"message": "Malformed input request"}, + {"x-amzn-errortype": "ValidationException"}, + ) + ) + + content, error = bedrock_harness.call_bedrock() + assert content is None + assert error is not None and "Bedrock request failed" in error + assert bedrock_harness.requests_sent == 1 + assert bedrock_harness.slept == [] + + +def test_bedrock_converse_transport_respects_retry_after_and_max_delay_cap( + bedrock_harness: _BedrockTransportHarness, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify _call_bedrock honors Retry-After within max_delay and aborts immediately when Retry-After exceeds max_delay.""" + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "3") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_BASE_DELAY", "0.1") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "5.0") + + retry_after_header = "2" + bedrock_harness.set_handler( + lambda req: ( + bedrock_harness.throttled_response(req, retry_after=retry_after_header) + if bedrock_harness.requests_sent == 1 + else bedrock_harness.ok_response(req, "ok") + ) + ) + + content, error = bedrock_harness.call_bedrock() + assert (content, error) == ("ok", None) + assert bedrock_harness.requests_sent == 2 + assert len(bedrock_harness.slept) == 1 + assert 2.0 <= bedrock_harness.slept[0] <= 2.6 + + # Exceed max_delay (3600s > 5.0s): must abort after 1 request without sleeping + bedrock_harness.requests_sent = 0 + bedrock_harness.slept.clear() + retry_after_header = "3600" + content, error = bedrock_harness.call_bedrock() + assert content is None + assert error is not None and "Bedrock request failed" in error + assert bedrock_harness.requests_sent == 1 + assert bedrock_harness.slept == [] + + +def test_bedrock_converse_transport_retries_transient_connection_error( + bedrock_harness: _BedrockTransportHarness, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify _call_bedrock retries transient botocore EndpointConnectionError up to SKILL_EVAL_LLM_MAX_RETRIES.""" + import botocore.exceptions + + monkeypatch.setenv("AWS_MAX_ATTEMPTS", "1") + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "2") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_BASE_DELAY", "0.05") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "0.2") + + def _conn_then_ok(req: Any) -> Any: + if bedrock_harness.requests_sent == 1: + raise botocore.exceptions.EndpointConnectionError(endpoint_url=req.url) + return bedrock_harness.ok_response(req, "reconnected") + + bedrock_harness.set_handler(_conn_then_ok) + + content, error = bedrock_harness.call_bedrock() + assert (content, error) == ("reconnected", None) + assert bedrock_harness.requests_sent == 2 + assert len(bedrock_harness.slept) == 1 + + +@pytest.mark.parametrize("exc_cls", [FileNotFoundError, PermissionError, IsADirectoryError, NotADirectoryError]) +def test_bedrock_converse_does_not_retry_filesystem_os_error( + bedrock_harness: _BedrockTransportHarness, + monkeypatch: pytest.MonkeyPatch, + exc_cls: type[OSError], +) -> None: + """Verify _call_bedrock fails fast without retrying when local credential file resolution raises filesystem OSError.""" + from unittest.mock import MagicMock + + import boto3 + + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "3") + calls = 0 + client = MagicMock() + + def raise_fs_error(**_kwargs: Any) -> None: + nonlocal calls + calls += 1 + raise exc_cls("missing or unreadable AWS credential file") + + client.converse.side_effect = raise_fs_error + monkeypatch.setattr(boto3, "client", lambda *_args, **_kwargs: client) + + content, error = bedrock_harness.call_bedrock() + assert content is None + assert error is not None and "Bedrock request failed" in error + assert calls == 1 + assert bedrock_harness.slept == [] + + +def test_bedrock_converse_aborts_when_judge_deadline_exhausted_before_retry( + bedrock_harness: _BedrockTransportHarness, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify _call_bedrock aborts without sleeping when _ACTIVE_JUDGE_DEADLINE is exceeded by the retry backoff.""" + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "3") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_BASE_DELAY", "1.0") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "30.0") + bedrock_harness.set_handler(lambda req: bedrock_harness.throttled_response(req, retry_after="5")) + + verifier = bedrock_harness.verifier + token = verifier._ACTIVE_JUDGE_DEADLINE.set(verifier.time.monotonic() + 0.5) + try: + content, error = bedrock_harness.call_bedrock() + finally: + verifier._ACTIVE_JUDGE_DEADLINE.reset(token) + + assert content is None + assert error is not None and "LLM judge time budget exhausted before retry" in error + assert bedrock_harness.requests_sent == 1 + assert bedrock_harness.slept == [] + + +@pytest.mark.parametrize( + ("raw_val", "expected_budget"), + [ + (None, 180.0), + ("", 180.0), + ("45", 45.0), + ("12.5", 12.5), + ("0", 180.0), + ("-10", 180.0), + ("nan", 180.0), + ("inf", 180.0), + ("-inf", 180.0), + ("not-a-number", 180.0), + ], +) +def test_resolve_judge_wall_time_budget_from_env( + harbor_eval_template: Any, + monkeypatch: pytest.MonkeyPatch, + raw_val: str | None, + expected_budget: float, +) -> None: + """Verify _resolve_judge_wall_time_budget parses positive finite seconds and falls back to 180.0 on invalid inputs.""" + if raw_val is None: + monkeypatch.delenv("SKILL_EVAL_LLM_JUDGE_BUDGET_SEC", raising=False) + else: + monkeypatch.setenv("SKILL_EVAL_LLM_JUDGE_BUDGET_SEC", raw_val) + + assert harbor_eval_template._resolve_judge_wall_time_budget() == pytest.approx(expected_budget) + + +def test_call_required_judge_enforces_skill_eval_llm_judge_budget_sec( + harbor_eval_template: Any, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify _call_required_judge applies the SKILL_EVAL_LLM_JUDGE_BUDGET_SEC deadline to judge execution.""" + monkeypatch.setenv("SKILL_EVAL_LLM_JUDGE_BUDGET_SEC", "42.5") + observed_remaining: list[float] = [] + + def fake_judge() -> dict[str, Any]: + observed_remaining.append(harbor_eval_template._remaining_judge_timeout(90.0)) + return {"score": 1.0, "reason": "ok"} + + result = harbor_eval_template._call_required_judge("accuracy", fake_judge) + assert result["score"] == 1.0 + assert len(observed_remaining) == 1 + assert 40.0 <= observed_remaining[0] <= 42.5 diff --git a/tests/test_tier2_command_failures.py b/tests/test_tier2_command_failures.py index bc33bad42..45ee6050f 100644 --- a/tests/test_tier2_command_failures.py +++ b/tests/test_tier2_command_failures.py @@ -245,6 +245,9 @@ def log_message(self, _format, *_args): monkeypatch.setenv(f"SKILL_EVAL_{purpose}_BASE_URL", f"http://127.0.0.1:{server.server_port}/v1") monkeypatch.setenv(f"SKILL_EVAL_{purpose}_API_KEY", "local-test-key") monkeypatch.setenv(f"SKILL_EVAL_{purpose}_MODEL", f"local-{purpose.lower()}-model") + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "2") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_BASE_DELAY", "0") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "0") try: invocation = CliRunner().invoke(cli, ["tier2", str(skill), "-o", str(reports)]) finally: @@ -254,7 +257,7 @@ def log_message(self, _format, *_args): assert requests[0][0] == "/v1/embeddings" chat_requests = [body for path, body in requests if path == "/v1/chat/completions"] - assert len(chat_requests) == 6 + assert len(chat_requests) == 6 * (3 if chat_status == 429 else 1) assert all(body["model"] == "local-llm-model" and body["stream"] is False for body in chat_requests) payload = json.loads((reports / "skillevaluator-tier2.json").read_text()) html = (reports / "skillevaluator-tier2.html").read_text() diff --git a/tests/test_tier3_public_runtime.py b/tests/test_tier3_public_runtime.py index c841dac1a..7608103ae 100644 --- a/tests/test_tier3_public_runtime.py +++ b/tests/test_tier3_public_runtime.py @@ -1290,3 +1290,27 @@ def handle_request(request: httpx.Request) -> httpx.Response: assert sdk_urls == [expected_url] assert verifier._anthropic_url() == expected_url + + +def test_write_task_toml_forwards_retry_env(tmp_path: Path) -> None: + """Verify that _write_task_toml stages LLM retry variables into [verifier.env].""" + case_dir = tmp_path / "case" + case_dir.mkdir() + runtime_env = { + "SKILL_EVAL_LLM_MAX_RETRIES": "5", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY": "2.0", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY": "40.0", + "UNRELATED_CUSTOM_VAR": "secret", + } + _write_task_toml( + case_dir, + {"id": "case-001", "expected_skill": "demo"}, + has_skill=True, + runtime_env=runtime_env, + ) + task = tomllib.loads((case_dir / "task.toml").read_text(encoding="utf-8")) + verifier_env = task["verifier"]["env"] + assert verifier_env["SKILL_EVAL_LLM_MAX_RETRIES"] == "${SKILL_EVAL_LLM_MAX_RETRIES}" + assert verifier_env["SKILL_EVAL_LLM_RETRY_BASE_DELAY"] == "${SKILL_EVAL_LLM_RETRY_BASE_DELAY}" + assert verifier_env["SKILL_EVAL_LLM_RETRY_MAX_DELAY"] == "${SKILL_EVAL_LLM_RETRY_MAX_DELAY}" + assert "UNRELATED_CUSTOM_VAR" not in verifier_env diff --git a/tests/test_verifier_error_redaction.py b/tests/test_verifier_error_redaction.py index acc3efec7..5dc18995b 100644 --- a/tests/test_verifier_error_redaction.py +++ b/tests/test_verifier_error_redaction.py @@ -5,7 +5,7 @@ from __future__ import annotations -import importlib.util +import email.message import io import json import sys @@ -13,14 +13,11 @@ from pathlib import Path import pytest +from tests.conftest import MockUrllibResponse, load_harbor_eval_template from skillevaluator.inference.client import LLMClient from skillevaluator.tier3.eval_core import llm_judge -_TEMPLATE = ( - Path(__file__).resolve().parents[1] / "src" / "skillevaluator" / "tier3" / "harbor" / "templates" / "eval.py" -) - _CREDENTIALS = { "OPENAI_API_KEY": "dummy-openai-credential-DO-NOT-USE", "NVIDIA_API_KEY": "dummy-nvidia-credential-DO-NOT-USE", @@ -38,6 +35,9 @@ *_CREDENTIALS, "SKILL_EVAL_LLM_PROVIDER", "SKILL_EVAL_LLM_BASE_URL", + "SKILL_EVAL_LLM_MAX_RETRIES", + "SKILL_EVAL_LLM_RETRY_BASE_DELAY", + "SKILL_EVAL_LLM_RETRY_MAX_DELAY", "OPENAI_BASE_URL", "ANTHROPIC_BASE_URL", "LLM_JUDGE_FALLBACK_MODELS", @@ -58,11 +58,8 @@ def _clean_provider_environment(monkeypatch: pytest.MonkeyPatch) -> None: @pytest.fixture def verifier_module(tmp_path: Path): module_name = f"harbor_template_error_redaction_{tmp_path.name}" - spec = importlib.util.spec_from_file_location(module_name, _TEMPLATE) - assert spec and spec.loader - module = importlib.util.module_from_spec(spec) + module = load_harbor_eval_template(module_name=module_name) sys.modules[module_name] = module - spec.loader.exec_module(module) module.VERIFIER_DIR = tmp_path module.REWARD_JSON = tmp_path / "reward.json" @@ -76,25 +73,11 @@ def _http_error(body: str, *, code: int = 401, reason: str = "Unauthorized") -> "https://provider.invalid/chat/completions", code, reason, - hdrs=None, + hdrs=email.message.Message(), fp=io.BytesIO(body.encode()), ) -class _FakeResponse: - def __init__(self, body: dict) -> None: - self._body = json.dumps(body).encode() - - def __enter__(self): - return self - - def __exit__(self, exc_type, exc, traceback) -> bool: - return False - - def read(self) -> bytes: - return self._body - - @pytest.mark.parametrize("module_fixture", ["source_judge", "verifier_module"]) def test_http_error_formatter_redacts_every_configured_credential( module_fixture: str, @@ -256,6 +239,7 @@ def test_generated_provider_error_preserves_selected_judge_provenance( def time_out(*_args, **_kwargs): raise TimeoutError(f"request timed out with {credential}") + monkeypatch.setattr(verifier_module.time, "sleep", lambda _s: None) monkeypatch.setattr(verifier_module.urllib.request, "urlopen", time_out) content, error, provenance = verifier_module._call_public_llm_with_provenance( @@ -340,7 +324,7 @@ def test_generated_call_public_llm_does_not_redact_successful_content( credential = _CREDENTIALS["OPENAI_API_KEY"] monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") monkeypatch.setenv("OPENAI_API_KEY", credential) - response = _FakeResponse({"choices": [{"message": {"content": f"model content: {credential}"}}]}) + response = MockUrllibResponse({"choices": [{"message": {"content": f"model content: {credential}"}}]}) monkeypatch.setattr(verifier_module.urllib.request, "urlopen", lambda *_args, **_kwargs: response) content, error = verifier_module.call_public_llm("safe prompt") @@ -451,3 +435,185 @@ def test_write_reward_outputs_preserves_fixed_schema_keys_when_a_credential_matc "security": 1.0, "overall": 0.75, } + + +@pytest.mark.parametrize( + ("status_code", "reason", "error_body"), + [ + (429, "Too Many Requests", "rate limited"), + (503, "Service Unavailable", "service unavailable"), + ], +) +def test_generated_call_public_llm_retries_transient_http_error_and_succeeds( + verifier_module, + monkeypatch: pytest.MonkeyPatch, + status_code: int, + reason: str, + error_body: str, +) -> None: + """Verify call_public_llm retries on transient HTTP status codes and succeeds on subsequent attempt.""" + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") + monkeypatch.setenv("OPENAI_API_KEY", "test-key") + + slept: list[float] = [] + monkeypatch.setattr(verifier_module.time, "sleep", slept.append) + + calls = 0 + + def mock_urlopen(*_args, **_kwargs): + nonlocal calls + calls += 1 + if calls == 1: + raise _http_error(error_body, code=status_code, reason=reason) + return MockUrllibResponse({"choices": [{"message": {"content": "judge success"}}]}) + + monkeypatch.setattr(verifier_module.urllib.request, "urlopen", mock_urlopen) + + content, error = verifier_module.call_public_llm("safe prompt") + + assert content == "judge success" + assert error is None + assert calls == 2 + assert len(slept) == 1 + + +def test_generated_call_public_llm_exhausts_retries_on_persistent_429( + verifier_module, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify call_public_llm exhausts retries and returns redacted error on persistent 429.""" + credential = _CREDENTIALS["OPENAI_API_KEY"] + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") + monkeypatch.setenv("OPENAI_API_KEY", credential) + + slept: list[float] = [] + monkeypatch.setattr(verifier_module.time, "sleep", slept.append) + + calls = 0 + + def fail_request(*_args, **_kwargs): + nonlocal calls + calls += 1 + raise _http_error(f'{{"error": "rate limited with {credential}"}}', code=429, reason="Too Many Requests") + + monkeypatch.setattr(verifier_module.urllib.request, "urlopen", fail_request) + + content, error = verifier_module.call_public_llm("safe prompt", allow_model_fallback=False) + + assert content is None + assert error is not None + assert "HTTP 429: Too Many Requests" in error + assert credential not in error + assert "[REDACTED]" in error + assert calls == 4 # 1 initial + 3 retries + assert len(slept) == 3 + + +def test_generated_call_public_llm_respects_retry_after_header( + verifier_module, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify call_public_llm respects Retry-After header on HTTP 429.""" + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") + monkeypatch.setenv("OPENAI_API_KEY", "test-key") + + slept: list[float] = [] + monkeypatch.setattr(verifier_module.time, "sleep", slept.append) + + hdrs = email.message.Message() + hdrs["Retry-After"] = "4.0" + + calls = 0 + + def mock_urlopen(*_args, **_kwargs): + nonlocal calls + calls += 1 + if calls == 1: + err = urllib.error.HTTPError( + "https://provider.invalid/chat/completions", + 429, + "Too Many Requests", + hdrs=hdrs, + fp=io.BytesIO(b'{"error": "rate limit"}'), + ) + raise err + return MockUrllibResponse({"choices": [{"message": {"content": "finished"}}]}) + + monkeypatch.setattr(verifier_module.urllib.request, "urlopen", mock_urlopen) + + content, error = verifier_module.call_public_llm("safe prompt") + + assert content == "finished" + assert error is None + assert calls == 2 + assert len(slept) == 1 + assert 4.0 <= slept[0] <= 4.6 + + +def test_generated_call_public_llm_fails_fast_on_http_401( + verifier_module, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify call_public_llm fails fast immediately on HTTP 401 Unauthorized without retry or sleep.""" + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") + monkeypatch.setenv("OPENAI_API_KEY", "invalid-key") + + slept: list[float] = [] + monkeypatch.setattr(verifier_module.time, "sleep", slept.append) + + calls = 0 + + def fail_unauthorized(*_args, **_kwargs): + nonlocal calls + calls += 1 + raise _http_error('{"error": "invalid api key"}', code=401, reason="Unauthorized") + + monkeypatch.setattr(verifier_module.urllib.request, "urlopen", fail_unauthorized) + + content, error = verifier_module.call_public_llm("safe prompt", allow_model_fallback=False) + + assert content is None + assert error is not None + assert "HTTP 401: Unauthorized" in error + assert calls == 1 + assert len(slept) == 0 + + +def test_generated_call_public_llm_fails_fast_if_retry_after_exceeds_max_delay( + verifier_module, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Verify call_public_llm fails fast without sleeping if Retry-After exceeds max_delay.""" + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") + monkeypatch.setenv("OPENAI_API_KEY", "test-key") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "10.0") + + slept: list[float] = [] + monkeypatch.setattr(verifier_module.time, "sleep", slept.append) + + hdrs = email.message.Message() + hdrs["Retry-After"] = "3600" + + calls = 0 + + def fail_request(*_args, **_kwargs): + nonlocal calls + calls += 1 + err = urllib.error.HTTPError( + "https://provider.invalid/chat/completions", + 429, + "Too Many Requests", + hdrs=hdrs, + fp=io.BytesIO(b'{"error": "rate limit"}'), + ) + raise err + + monkeypatch.setattr(verifier_module.urllib.request, "urlopen", fail_request) + + content, error = verifier_module.call_public_llm("safe prompt", allow_model_fallback=False) + + assert content is None + assert error is not None + assert "HTTP 429: Too Many Requests" in error + assert calls == 1 + assert len(slept) == 0 diff --git a/tests/tier3/test_judge_failure_artifacts.py b/tests/tier3/test_judge_failure_artifacts.py index fbe8225f7..8a9a158f4 100644 --- a/tests/tier3/test_judge_failure_artifacts.py +++ b/tests/tier3/test_judge_failure_artifacts.py @@ -5,15 +5,19 @@ from __future__ import annotations +import asyncio import importlib.util import json import math import os import shutil +import signal import subprocess import sys +import time +import urllib.error from pathlib import Path -from types import ModuleType +from types import ModuleType, SimpleNamespace import pytest @@ -35,6 +39,7 @@ def _load_verifier(tmp_path: Path) -> ModuleType: + """Load and initialize the Harbor verifier template module in a temporary workspace.""" module_name = f"harbor_eval_failure_artifacts_{tmp_path.name}" spec = importlib.util.spec_from_file_location(module_name, _EVAL_TEMPLATE) assert spec and spec.loader @@ -92,6 +97,7 @@ def test_verifier_main_fails_closed_after_collecting_every_required_judge( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: + """Verify verifier main fails closed with exit code 1 after collecting every required judge error.""" verifier = _load_verifier(tmp_path) credential = "dummy-secret-credential-DO-NOT-RETAIN" monkeypatch.setenv("ANTHROPIC_API_KEY", credential) @@ -157,10 +163,179 @@ def behavior_check(*_args, **_kwargs): assert overall_score(numeric) is None +def test_verifier_retries_leave_time_to_write_failure_artifacts( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Ensure slow transient calls finish before Harbor verifier timeout kills artifact writes.""" + verifier = _load_verifier(tmp_path) + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") + monkeypatch.setenv("OPENAI_API_KEY", "test-key-fake") + monkeypatch.setenv("SKILL_EVAL_LLM_MAX_RETRIES", "3") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_BASE_DELAY", "0") + monkeypatch.setenv("SKILL_EVAL_LLM_RETRY_MAX_DELAY", "0") + monkeypatch.setenv("LLM_JUDGE_FALLBACK_MODELS", "") + monkeypatch.setattr(verifier, "_ragas_goal_accuracy_enabled", lambda: False) + + elapsed = [0.0] + + class FakeTime: + """Simulate time progression for slow network timeouts.""" + + def monotonic(self) -> float: + return elapsed[0] + + def sleep(self, seconds: float) -> None: + elapsed[0] += seconds + + def __getattr__(self, name: str): + return getattr(time, name) + + def slow_timeout(_request, timeout=90): + elapsed[0] += timeout + raise urllib.error.URLError("timed out") + + monkeypatch.setattr(verifier, "time", FakeTime()) + monkeypatch.setattr(verifier.urllib.request, "urlopen", slow_timeout) + + with pytest.raises(SystemExit) as exc_info: + verifier.main() + + assert exc_info.value.code == 1 + assert elapsed[0] <= 540.0 + rich = json.loads(verifier.SKILL_EVALUATOR_REWARD_JSON.read_text(encoding="utf-8")) + numeric = json.loads(verifier.REWARD_JSON.read_text(encoding="utf-8")) + assert rich["evaluation_status"] == "failed" + assert "accuracy" in rich["evaluation_errors"] + assert numeric["overall"] == 0.0 + assert overall_score(numeric) is None + + +def test_required_judge_deadline_interrupts_a_stalled_response_body( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Interrupt stalled provider response body when required judge deadline expires.""" + if not hasattr(signal, "setitimer") or signal.getitimer(signal.ITIMER_REAL)[0] > 0: + pytest.skip("free POSIX interval timer required") + verifier = _load_verifier(tmp_path) + monkeypatch.setattr(verifier, "_JUDGE_WALL_TIME_BUDGET_SEC", 0.05) + + class SlowResponse: + """Simulate a trickling or stalled response body.""" + + def __enter__(self): + return self + + def __exit__(self, *_args): + return False + + def read(self): + time.sleep(0.5) + return b"late success" + + monkeypatch.setattr(verifier.urllib.request, "urlopen", lambda *_args, **_kwargs: SlowResponse()) + previous_handler = signal.getsignal(signal.SIGALRM) + started = time.monotonic() + result = verifier._call_required_judge("accuracy", lambda: verifier._urlopen_with_retry("test")) + + assert time.monotonic() - started < 0.4 + assert result["status"] == "error" + assert result["score"] is None + assert signal.getsignal(signal.SIGALRM) == previous_handler + assert signal.getitimer(signal.ITIMER_REAL)[0] == 0 + + +def test_required_judge_restores_alarm_handler_after_teardown_interrupt( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Restore previous SIGALRM handler when judge deadline teardown is interrupted.""" + if not hasattr(signal, "setitimer") or signal.getitimer(signal.ITIMER_REAL)[0] > 0: + pytest.skip("free POSIX interval timer required") + verifier = _load_verifier(tmp_path) + original_setitimer = signal.setitimer + previous_handler = signal.getsignal(signal.SIGALRM) + + def interrupted_setitimer(timer, seconds, interval=0): + previous = original_setitimer(timer, seconds, interval) + if seconds == 0: + raise TimeoutError("interrupted during deadline teardown") + return previous + + monkeypatch.setattr(verifier.signal, "setitimer", interrupted_setitimer) + result = verifier._call_required_judge("accuracy", lambda: {"score": 1.0, "reason": "ok"}) + + assert result["status"] == "error" + assert signal.getsignal(signal.SIGALRM) == previous_handler + assert signal.getitimer(signal.ITIMER_REAL)[0] == 0 + + +def test_ragas_goal_judge_obeys_the_required_judge_time_budget(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Ensure Ragas goal scorer returns before its Harbor budget expires.""" + verifier = _load_verifier(tmp_path) + monkeypatch.setenv("SKILL_EVAL_LLM_PROVIDER", "openai") + monkeypatch.setenv("OPENAI_API_KEY", "test-key-fake") + monkeypatch.delenv("OPENAI_BASE_URL", raising=False) + monkeypatch.delenv("SKILL_EVAL_LLM_BASE_URL", raising=False) + monkeypatch.setattr(verifier, "_ragas_goal_accuracy_enabled", lambda: True) + monkeypatch.setattr(verifier, "_JUDGE_WALL_TIME_BUDGET_SEC", 0.01) + + class FakeMessage: + """Mock message object for Ragas sample input.""" + + def __init__(self, content: str): + self.content = content + + class FakeMetric: + """Mock metric object for Ragas async evaluation.""" + + def __init__(self, llm): + self.llm = llm + + async def ascore(self, _sample): + await asyncio.sleep(0.05) + return SimpleNamespace(value=1.0) + + fake_ragas = ModuleType("ragas") + fake_ragas.SingleTurnSample = lambda **_kwargs: object() + fake_ragas_llms = ModuleType("ragas.llms") + fake_ragas_llms_base = ModuleType("ragas.llms.base") + fake_ragas_llms_base.llm_factory = lambda *_args, **_kwargs: object() + fake_ragas_messages = ModuleType("ragas.messages") + fake_ragas_messages.AIMessage = FakeMessage + fake_ragas_messages.HumanMessage = FakeMessage + fake_ragas_metrics = ModuleType("ragas.metrics") + fake_ragas_collections = ModuleType("ragas.metrics.collections") + fake_ragas_collections.AgentGoalAccuracyWithReference = FakeMetric + fake_openai = ModuleType("openai") + fake_openai.AsyncOpenAI = lambda **_kwargs: object() + for name, module in { + "ragas": fake_ragas, + "ragas.llms": fake_ragas_llms, + "ragas.llms.base": fake_ragas_llms_base, + "ragas.messages": fake_ragas_messages, + "ragas.metrics": fake_ragas_metrics, + "ragas.metrics.collections": fake_ragas_collections, + "openai": fake_openai, + }.items(): + monkeypatch.setitem(sys.modules, name, module) + monkeypatch.setattr( + verifier.urllib.request, + "urlopen", + lambda *_args, **_kwargs: pytest.fail("custom fallback must stop before another HTTP request"), + ) + + result = verifier._call_required_judge( + "goal_accuracy", verifier.judge_goal_accuracy, "question", "ground truth", "agent response" + ) + + assert result["status"] == "error" + assert result["score"] is None + + def test_verifier_main_keeps_genuine_zero_judge_verdicts_scoreable( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: + """Retain genuine zero-score verdicts from successful judge evaluations as scoreable metrics.""" verifier = _load_verifier(tmp_path) calls: list[str] = [] @@ -198,6 +373,7 @@ def test_verifier_main_recovers_malformed_accuracy_and_goal_judges( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: + """Recover malformed accuracy and goal judge responses on retry and record overall score.""" verifier = _load_verifier(tmp_path) monkeypatch.setattr(verifier, "_ragas_goal_accuracy_enabled", lambda: False) pair_calls: list[tuple[str, dict]] = [] @@ -262,6 +438,7 @@ def test_verifier_retries_non_string_judge_text_before_collector_and_report( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: + """Retry non-string judge text payloads before collector and report generation.""" verifier = _load_verifier(tmp_path) monkeypatch.setattr(verifier, "_ragas_goal_accuracy_enabled", lambda: False) accuracy_calls: list[str] = [] @@ -352,6 +529,7 @@ def test_report_coerces_and_bounds_non_string_reasons_from_existing_artifacts( score: float, detail: dict, ) -> None: + """Coerce and bound non-string reason fields from legacy judge artifacts.""" reward = { "entry_id": "legacy-judge-artifact", metric: score, @@ -367,6 +545,7 @@ def test_report_coerces_and_bounds_non_string_reasons_from_existing_artifacts( def test_report_redacts_configured_secret_before_bounding_reason(monkeypatch: pytest.MonkeyPatch) -> None: + """Redact configured secrets before truncating long judge reason strings in report.""" credential = "SECRET-ABCDEFGHIJKLMNOPQRSTUVWXYZ-0123456789" prefix = "x" * 490 monkeypatch.setenv("OPENAI_API_KEY", credential) @@ -388,6 +567,7 @@ def test_verifier_main_keeps_accuracy_fail_closed_after_retry_exhaustion( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, ) -> None: + """Keep accuracy metric fail-closed when verifier retries are exhausted.""" verifier = _load_verifier(tmp_path) credential = "dummy-verifier-retry-secret-DO-NOT-RETAIN" monkeypatch.setenv("NVIDIA_API_KEY", credential) @@ -439,6 +619,7 @@ def goal_call(prompt: str, **kwargs): def test_verifier_main_keeps_documented_neutral_judge_skips_scoreable( tmp_path: Path, ) -> None: + """Keep documented neutral judge skips scoreable with default passing scores.""" verifier = _load_verifier(tmp_path) entry = json.loads(verifier.ENTRY_PATH.read_text(encoding="utf-8")) entry["ground_truth"] = "" @@ -464,6 +645,7 @@ def test_verifier_main_normalizes_malformed_or_raised_judge_failures_and_continu monkeypatch: pytest.MonkeyPatch, failure_kind: str, ) -> None: + """Normalize malformed or raised judge failures and continue evaluating remaining judges.""" verifier = _load_verifier(tmp_path) calls: list[str] = [] @@ -503,6 +685,7 @@ def judge(*_args, **_kwargs): def test_numeric_reward_payload_excludes_boolean_and_non_finite_values(tmp_path: Path) -> None: + """Exclude boolean, non-finite, and infinite values from numeric reward payloads.""" verifier = _load_verifier(tmp_path) payload = verifier._numeric_reward_payload( @@ -523,6 +706,7 @@ def test_numeric_reward_payload_excludes_boolean_and_non_finite_values(tmp_path: def test_evaluation_failure_fields_are_reserved_metadata() -> None: + """Confirm evaluation failure fields are reserved metadata in custom grader runner.""" expected = {"evaluation_status", "evaluation_errors"} assert expected <= RESERVED_METRIC_NAMES @@ -535,6 +719,7 @@ def test_evaluation_failure_fields_are_reserved_metadata() -> None: def _run_generated_test_sh(task_dir: Path, env: dict[str, str]) -> subprocess.CompletedProcess[str]: + """Execute the generated test.sh script with the specified environment variables.""" return subprocess.run( ["bash", str(task_dir / "tests" / "test.sh")], check=False, @@ -549,6 +734,7 @@ def test_generated_standard_grading_scripts_stop_after_evaluator_failure( tmp_path: Path, grading_mode: str, ) -> None: + """Halt generated grading scripts immediately when standard evaluator fails.""" task_dir = tmp_path / grading_mode _write_test_sh(task_dir, grading_mode=grading_mode, custom_grader=grading_mode == "default_plus_custom") tests_dir = task_dir / "tests" @@ -566,6 +752,7 @@ def test_generated_standard_grading_scripts_stop_after_evaluator_failure( def test_generated_custom_only_script_accepts_overall_only_custom_reward(tmp_path: Path) -> None: + """Accept overall-only reward payloads from custom grading scripts.""" task_dir = tmp_path / "custom-only" _write_test_sh(task_dir, grading_mode="custom_only", custom_grader=True) tests_dir = task_dir / "tests" diff --git a/tests/tier3/test_judge_parse_robustness.py b/tests/tier3/test_judge_parse_robustness.py index 74647b96c..5bd9fa3f5 100644 --- a/tests/tier3/test_judge_parse_robustness.py +++ b/tests/tier3/test_judge_parse_robustness.py @@ -653,6 +653,23 @@ def test_template_native_openai_gpt5_payload_matches_eval_core(model): assert "max_tokens" not in template_payload +def test_template_schema_and_payload_match_eval_core(): + assert eval_template.ACCURACY_JSON_SCHEMA == llm_judge.ACCURACY_JSON_SCHEMA + assert eval_template.GOAL_ACCURACY_JSON_SCHEMA == llm_judge.GOAL_ACCURACY_JSON_SCHEMA + assert eval_template.BEHAVIOR_CHECK_JSON_SCHEMA == llm_judge.BEHAVIOR_CHECK_JSON_SCHEMA + kwargs = { + "model": "gpt-4o", + "prompt": "Judge this response", + "max_tokens": 512, + "temperature": 0.0, + "provider": "openai", + "request_url": llm_judge.OPENAI_CHAT_URL, + "response_schema": llm_judge.ACCURACY_JSON_SCHEMA, + "schema_name": "accuracy_judgment", + } + assert eval_template._chat_completion_payload(**kwargs) == llm_judge._chat_completion_payload(**kwargs) + + def test_template_salvage_score_matches_eval_core_on_partial_recovery(monkeypatch): # Drift guard for the salvage denominator fix: 3 salvaged of 7 expected # must score 3/7 in BOTH the shared judge and the in-sandbox template. diff --git a/tests/tier3/test_judge_result_semantics.py b/tests/tier3/test_judge_result_semantics.py index 5b8e18d8d..070baee8d 100644 --- a/tests/tier3/test_judge_result_semantics.py +++ b/tests/tier3/test_judge_result_semantics.py @@ -380,7 +380,9 @@ def test_shared_structured_judge_keeps_explicit_max_tokens_override( assert result["score"] is not None assert len(calls) == 1 - assert calls[0]["kwargs"] == {"max_tokens": 2048} + assert calls[0]["kwargs"]["max_tokens"] == 2048 + assert isinstance(calls[0]["kwargs"]["response_schema"], dict) + assert isinstance(calls[0]["kwargs"]["schema_name"], str) def test_judge_error_reserved_fields_cannot_be_overridden(judge_module) -> None: