[ananthsub]

What does this PR do?

Counts every dataset row in the collation metrics, not only run requests and materialized rows.

#4060 counts materialized rows as examples and reports their task-input size. Any other row without a valid responses_create_params, such as a task row for an environment server with its own input in a dataset that does not declare taskset:, was still marked offending. Its dataset's metrics sidecar reported zero examples, and drift in it went undetected.

Where this fits

Datasets reach an environment server in one of two ways today: run request rows route by their agent, and datasets that declare taskset: are materialized during collation and route by environment_server_routes. The goal is one path, where every dataset declares a taskset and routes to an environment server:

flowchart LR
    P["Preparation script<br/>task rows with a stable task_id"] --> C["Collation<br/>validates, computes metrics,<br/>materializes"]
    Y["Dataset YAML<br/>file + taskset"] --> C
    S["task_data.py schemas<br/>environment server + resources server"] -.-> C
    C --> R["Rollout collection<br/>routes by taskset"]
    T["environment_server_routes"] --> R
    R --> E["Environment server<br/>single-agent, multi-agent,<br/>or legacy_agent relay"]

Loading

The UserSim stack (#3800 and #3802) has merged. UserSim's datasets already declare taskset: and commit plain task rows, so they use the one path. Each step is additive, and datasets without taskset: keep working until the last one:

flowchart TB
    S4053["PR 4053 (merged)<br/>tasksets route to servers<br/>with several agents"]
    S4060["PR 4060 (merged)<br/>metrics for<br/>materialized rows"]
    S3800["PR 3800 (merged)<br/>UserSim runtime"]
    S3802["PR 3802 (merged)<br/>UserSim validation tasks"]
    S4096["PR 4096<br/>stable, unique<br/>task IDs"]
    S4097["PR 4097<br/>metrics for<br/>every row"]
    S4098["PR 4098<br/>schema per<br/>environment server"]
    S4099["PR 4099<br/>legacy_agent relays<br/>serve tasksets"]
    SREF["Remove the bridge<br/>for already-materialized<br/>rows"]
    SNEXT["Later<br/>configs declare taskset:,<br/>then agent routing retires"]
    S4053 --> S4060 --> S3800 --> S3802 --> S4096 --> S4097 --> S4098 --> S4099 --> SREF --> SNEXT
    classDef done fill:#F1EFE8,stroke:#5F5E5A,color:#444441
    classDef prereq fill:#FAEEDA,stroke:#854F0B,color:#633806
    classDef this fill:#EEEDFE,stroke:#534AB7,color:#26215C,stroke-width:2px
    classDef next fill:#E1F5EE,stroke:#0F6E56,color:#085041
    class S4053,S4060,S3800,S3802 done
    class S4097 this
    class S4096,S4098,S4099,SREF,SNEXT next

Loading

Gray: merged. Purple: this PR. Green: other steps.

Base branch. This stack is based on main. Review only this PR's own diff.

Changes

• Every JSON object row is an example; only rows that are not JSON objects are offending. require_responses is gone, because the row, not the dataset's declaration, decides whether Responses API metrics apply.
• Run request rows keep their Responses API metrics, so their sidecars keep the same keys and values.
• Other rows report their size through feat(data): support materialized task dataset metrics #4060's Json-dumped task-input words (proxy for token count), so datasets have one size metric instead of two, and a change inside nested task fields is a metrics conflict like any other. The metric is written only for datasets that have such rows.
• Whether a row is valid for its environment server is decided by schema validation (feat(data): validate rows against their environment server's task schema #4098), not by metrics.
• prepare-validate.mdx and resources-server-apis.mdx describe the metrics.

No issue is linked. This follows #4060.

Validation

• tests/unit_tests/test_task_materialization.py: task rows without responses_create_params in a dataset without taskset: are counted, their task-input size is written, and a change in nested fields is a metrics conflict. Run request rows report no task metrics. These tests fail without the change.
• tests/unit_tests/test_train_data_utils.py: the offending-row tests use rows that are not JSON objects, and feat(data): support materialized task dataset metrics #4060's test for a row that is not a run request now expects its task-input size.
• Every committed dataset with a metrics sidecar was checked. Datasets routed by agent contain only run request rows, so their sidecars do not change. UserSim's example dataset declares taskset:, so feat(data): collate declared tasksets #3818 already counted its rows; it now also reports Json-dumped task-input words (proxy for token count). Drift detection accepts new keys, so gym dataset collate +mode=example_validation on its config neither conflicts with nor rewrites the committed sidecar.
• pytest on the core data, collation, and rollout test modules: all pass on this branch.
• pre-commit run on the changed files: all pass.

Rollouts

N/A. The change affects collation metrics only, and committed sidecars are unchanged.

Compatibility and benchmark impact

• Datasets whose rows are all run requests produce the same sidecars.
• A dataset without taskset: whose rows are not run requests now reports their count and size instead of zero examples. No committed dataset has such rows.
• A dataset with taskset: and such rows, such as UserSim's, gains the size metric as a new key. That is not a conflict with its committed sidecar; regenerating the sidecar adds the key.
• No benchmark impact.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.