[Feat] Opt-in screenshot input for hosted Jev (S1A_JEV_IMAGES, TypeSafe danger zone)

CHANGELOG.md

@@ -14,6 +14,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); ver
 
 ### Added
 
+- Experimental (TypeSafe danger zone): `S1A_JEV_IMAGES=1` lets `--model jev` on the direct TypeSafe backend read one
+  current screenshot (TypeSafe `assets` plus the `TypeSafe-Danger-Zone-Images` header), resized or rejected to the
+  documented limits before sending, and enables `--pixel-target` for `--model jev`. Off by default; the proxy
+  backend refuses it. Step records gain `sent_image`. No live comparison is recorded yet (#97).
 - `s1a probe --order reverse|shuffle:N` (with `--seed` and `--json PATH`): the option-order check. Every case is
   asked in file order and again with the same keys and texts in another dict order; the probe reports the flip
   rate, accuracy in each order and a row per case with every order asked. Verdict and exit code are unchanged

docs/capabilities.md

@@ -10,7 +10,7 @@ For issue trackers and serving support, see [System1-Omni model tracker #83](htt
 
 | Model / checkpoint | Input modality | Question types implemented | Scenario / front | Serving path | Hardware validation in this repository |
 |---|---|---|---|---|---|
-| TypeSafe Jev (`--model jev`; actual served model comes from the configured endpoint) | Text state and enumerated options | `choice`, `noul`; service protocol also defines `score`, but Agents has no corresponding question type | Tool decisions, browser decisions, rails, `decide`/`probe` and MCP | HTTP through TypeSafe or configured OpenRouter-compatible endpoint; configured compatible endpoint can use System1-Omni `/v1/systemone` | The adapter can address configured Jev-compatible services; exact model identity is endpoint-reported. Historical artifacts include Open-Jev-9B and an Open-Jev-27B-v1.1 load attempt; those are endpoint configurations, not model names guaranteed by the `jev` adapter. See below. Other combinations **unverified here.** |
+| TypeSafe Jev (`--model jev`; actual served model comes from the configured endpoint) | Text state and enumerated options; with `S1A_JEV_IMAGES=1` on direct TypeSafe, also one current screenshot (**experimental, TypeSafe danger zone**: offline fake-transport tests only, no recorded live run or quality result, #97) | `choice`, `noul`; service protocol also defines `score`, but Agents has no corresponding question type | Tool decisions, browser decisions, rails, `decide`/`probe` and MCP | HTTP through TypeSafe or configured OpenRouter-compatible endpoint; configured compatible endpoint can use System1-Omni `/v1/systemone` | The adapter can address configured Jev-compatible services; exact model identity is endpoint-reported. Historical artifacts include Open-Jev-9B and an Open-Jev-27B-v1.1 load attempt; those are endpoint configurations, not model names guaranteed by the `jev` adapter. See below. Other combinations **unverified here.** |
 | JEV-27B-VL (`--model jev-vl-served`; `autotrust/JEV-27B-VL`) | Exactly one current PNG/JPEG plus text state and question context | `choice`, `noul` | Browser front and direct `build_model()` factory path; other fronts are not registered | HTTP native flat `/v1/systemone` through the System1-Omni frontend at `JEV_VL_URL` | Local adapter/registration tests use mocks. Single-H800 GPU checks and bounded R3 browser task completion with same-trial video are validated; other tasks and hardware remain unverified. See [validation scope and limits](decision-models.md#served-jev-vl). |
 | CLM-v0.1-8B (`--model clm`; default serving label `clm-latest`) | Text | `choice`, `noul` | Tool decisions, browser, rails, `decide`/`probe`, MCP | HTTP `/v1/systemone` to a compatible CLM service; real encoder is external | Adapter fixtures include stub responses. A historical ticket-router report records real Qwen3-8B through upstream `clm-serve` on an RTX 4090, but its Agents source SHA is not pinned in the report and the serving engine is not Omni native CLM. It reports 18/90 correct (20%) for the normal tool-loop framing and a 4/12 fit-probe verdict of “not a decision-model task”; the 90 decisions repeat the same 30-ticket set and are not 90 independent tickets. Treat this as poor, provenance-incomplete historical evidence, not current or broad validation. See [CLM report](../evals/ticket_router/CLM.md). |
 | LAYA (`--model laya`; default `convaiinnovations/laya`, English; `LAYA_SUBFOLDER=multilingual` selects multilingual) | Text | `choice`, `noul` | Tool, browser, rails, `decide`/`probe`; browser state/question shaping is implemented | In-process Python `laya` | Historical English and multilingual CPU/MPS runs are documented below. Other/typed/custom checkpoints and CUDA are unverified here. |

docs/configuration.md

@@ -11,6 +11,7 @@ Variables can be exported in your shell or placed in a `.env` file at the root o
 | `TYPESAFE_API_KEY` | `jev` model | *(unset)* | API key for direct TypeSafe decisions endpoint (`https://api.typesafe.ai/v1/systemone`). |
 | `TYPESAFE_API_URL` | `jev` model | `https://openrouter.ai/api/alpha/decisions` | Endpoint URL for decisions; defaults to OpenRouter proxy, or can be overridden to a custom proxy URL. |
 | `TYPESAFE_MODEL` | `jev` model | `jev-1.13.0` direct, `typesafe/jev-1.13` through OpenRouter | Jev model id on either backend. Both defaults are pinned versions: no path sends the moving `jev-latest` or `jev-preview` aliases unless you set them here. On the direct path (`TYPESAFE_API_KEY` without `TYPESAFE_API_URL`) it takes a TypeSafe name such as `jev-1.14.0`; an OpenRouter id like `typesafe/jev-1.13` there is a config error. TypeSafe keeps `jev-1.13.0` available and promises 60 days' notice before deprecating it. Step records keep `model` (the id the response reports, else the one sent) and `requested_model` (the one sent). |
+| `S1A_JEV_IMAGES` | `jev` model (direct TypeSafe only) | *(unset: off)* | **Experimental (TypeSafe danger zone).** `1`/`true` lets hosted Jev read the observation's screenshot: the browser front then captures one per step, and the desktop `--pixel-target` path accepts `--model jev`. Each request carries one PNG/JPEG in TypeSafe's `assets` map, referenced once from `state.screenshot` (a fresh UUID reference per request), plus the `TypeSafe-Danger-Zone-Images: true` header. Screenshots whose longest side is over 1,568 px are scaled down (aspect kept, sent as PNG); a side under 280 px after that, a file over 16 MiB or a body over 32 MiB is a config error before anything is sent. Step records add `sent_image` (sent width, height, bytes, format, whether it was resized); image tokens are billed as `input_tokens`. Set with the OpenRouter proxy or `TYPESAFE_API_URL` it is a config error. An image request can take longer than the 5 s default deadline; raise `S1A_DECISION_TIMEOUT_S` if it times out. |
 | `S1A_DECISION_TIMEOUT_S` | `jev`, `jev-vl-served` models | `5` | Total deadline in seconds for one decision. Jev includes retries; JEV-VL shares it across sequential questions and does not retry HTTP requests. Must be finite and above zero. Raise it for a slower System One server behind `TYPESAFE_API_URL` (e.g. a local OneJev-27B or CLM-8B: `30`). |
 | `JEV_VL_URL` | `jev-vl-served` model | *(unset, required)* | Frontend base URL serving `autotrust/JEV-27B-VL` through System1-Omni's native flat `/v1/systemone` protocol, e.g. `http://127.0.0.1:8080`. Browser and direct factory paths only; requires exactly one current PNG/JPEG image. |
 | `OPENROUTER_API_KEY` | `jev` model (proxy), chat model fallback | *(unset)* | OpenRouter API key, used for proxying Jev decisions or as a fallback for `LLM_API_KEY`. |

docs/decision-models.md

@@ -58,7 +58,7 @@ shorthands; `warm()` and `close()` open and release the backend.
 
 | `--model` | class | `name` | notes |
 |---|---|---|---|
-| `jev` | `JevModel(transport)` | `jev` | the request body every front sent before the layer existed, byte for byte; `from_env` picks TypeSafe or the OpenRouter proxy (see [configuration.md](configuration.md)) |
+| `jev` | `JevModel(transport)` | `jev` | the request body every front sent before the layer existed, byte for byte; `from_env` picks TypeSafe or the OpenRouter proxy (see [configuration.md](configuration.md)). Text only unless `S1A_JEV_IMAGES=1` on the direct TypeSafe backend (**experimental, TypeSafe danger zone**, #97): then one current screenshot rides in `assets`, referenced from `state.screenshot`, with the danger-zone header; no quality result is claimed for it yet |
 | `jev-vl-served` | `ServedJevVLModel(url=, timeout_s=)` | `jev-vl-served` | `autotrust/JEV-27B-VL` through native flat `POST /v1/systemone`; exactly one current PNG/JPEG, choice and noul only; sequential questions share one deadline, with no HTTP retry or answer re-ask. Browser and direct factory paths; local tests use mocks, with separate [bounded R3 GPU/browser validation](#served-jev-vl). |
 | `clm` | `ClmModel(client, model=)` | `clm` | CLM behind `/v1/systemone`: a frozen Qwen3-8B encoder and two projection heads, scored by an engine that owns everything after the encoder. One POST per decision; `MODEL_SERVICE_CONFIG_ERROR` without `CLM_URL`, `MODEL_CALL_FAILED` when nothing is listening, on an error status, or on a body with no `answers`. Records `url`, the `X-Request-Id` it sent and `served_by` (the served names, read once — by `warm()` on the agent fronts, or on the first decision for `decide`/`probe`); choice and noul ([docs/clm.md](clm.md)) |
 | `laya` | `LayaModel(agent, model=)` | `laya` | one forward pass per call on a thread; `MODEL_SERVICE_CONFIG_ERROR` when `input_tokens` fills the window (Laya cuts the state silently; see `LAYA_MAX_LEN` and `LAYA_HEAD_MAX_LEN` in [configuration.md](configuration.md)); `ValueError` and `RuntimeError` from the library become `MODEL_CALL_FAILED` |

s1a/agents/desktop.py

@@ -26,6 +26,7 @@
 from pathlib import Path
 from typing import Any, AsyncIterator
 
+from s1a.decision_models.wire import decisions_backend_from_env, jev_images_from_env
 from s1a.desktop.driver import CuaDriver, DriverError, Snapshot, driver_from_env, opened
 from s1a.desktop.env import ABSTAIN, WindowEnv, clickable, observable
 from s1a.spec import Budget, Series, ToolAgentSpec, positive_float, positive_int
@@ -119,10 +120,14 @@ def make_series(flags: argparse.Namespace) -> Series:
         and os.getenv("CUA_S1_VARIANT", "nano") == "4b"
         and os.getenv("CUA_S1_MODALITY", "multimodal") == "multimodal"
     )
+    if pixel_targets and flags.model == "jev":
+        # experimental hosted Jev screenshots (#97): a config error on the proxy backend, False when unset
+        visual_model = jev_images_from_env(decisions_backend_from_env())
     if pixel_targets and flags.model not in {"rule", "random"} and not visual_model:
         raise ValueError(
             "--pixel-target requires a screenshot model: use --model cua with CUA_S1_VARIANT=4b and "
-            "CUA_S1_MODALITY=multimodal, or an explicit rule/random baseline"
+            "CUA_S1_MODALITY=multimodal, --model jev with S1A_JEV_IMAGES=1 on the direct TypeSafe backend "
+            "(experimental), or an explicit rule/random baseline"
         )
     plan = parse_plan(flags.plan) if flags.plan else ()
     if flags.verify_file and not flags.text:

s1a/decision_models/base.py

@@ -37,6 +37,14 @@ async def decide(self, body: Json) -> tuple[Json, int]: ...
     async def close(self) -> None: ...
 
 
+class JevImageTransport(JevTransport, Protocol):
+    """A transport that can carry screenshots (``images`` True): image bodies pass per-request ``headers``."""
+
+    images: bool
+
+    async def decide(self, body: Json, *, headers: dict[str, str] | None = None) -> tuple[Json, int]: ...
+
+
 class DecisionModel(ABC):
     """A model that reads an observation and a discrete action space and returns a distribution over it."""
 

s1a/decision_models/fakes.py

@@ -16,6 +16,7 @@ class ScriptedTransport:
     """
 
     model = "typesafe/jev-test"
+    images = False  # set True to stand in for a TypeSafe client with S1A_JEV_IMAGES on
 
     def __init__(
         self,
@@ -34,12 +35,14 @@ def __init__(
         self._usage = usage or {"input_tokens": 0, "output_tokens": 0}
         self._latency_ms = latency_ms
         self.bodies: list[Json] = []
+        self.headers: list[dict[str, str]] = []  # per call; empty when the call passed none
 
     async def warm(self) -> None:
         return None
 
-    async def decide(self, body: Json) -> tuple[Json, int]:
+    async def decide(self, body: Json, *, headers: dict[str, str] | None = None) -> tuple[Json, int]:
         self.bodies.append(body)
+        self.headers.append(dict(headers or {}))
         if self._error is not None:
             raise self._error
         if self._answers: