Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions fern/versions/latest/pages/data/prepare-validate.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,37 @@ This validates your data and adds the `agent_ref` field to each row, routing sam

---

## Download Missing Remote Datasets

If a dataset in your config has a remote identifier, `ng_prepare_data` can download it before validation. Use `+should_download=true` and select the matching `+data_source`.

For configs that use `gitlab_identifier`, such as the SWE agents `swebench_openhands.yaml` validation dataset, use the GitLab registry backend:

```bash
config_paths="responses_api_agents/swe_agents/configs/swebench_openhands.yaml,\
responses_api_models/vllm_model/configs/vllm_model.yaml"

ng_prepare_data "+config_paths=[$config_paths]" \
+output_dirpath=data/swebench_openhands \
+mode=train_preparation \
+should_download=true \
+data_source=gitlab
```

The same dataset can also be downloaded directly:

```bash
ng_download_dataset_from_gitlab \
+dataset_name=swebench_verified_for_sweagent_and_openhands \
+version=0.0.1 \
+artifact_fpath=swebench_verified_for_sweagent_and_openhands.jsonl \
+output_fpath=responses_api_agents/swe_agents/data/swebench_verified_for_sweagent_and_openhands.jsonl
```

Use `ng_download_dataset_from_hf` only for configs that use `huggingface_identifier`. A 401 from the Hugging Face downloader usually means the command is pointing at the wrong registry or you need access to a gated/private Hugging Face repo.

---

## Validation Modes

| Mode | Purpose | Validates |
Expand Down
10 changes: 10 additions & 0 deletions fern/versions/latest/pages/reference/cli-commands.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,16 @@ ng_prepare_data "+config_paths=[${config_paths}]" \
+mode=example_validation
```

To download missing datasets from a config before validation, add `+should_download=true` and set `+data_source` to the registry used by that dataset's identifier. For example, configs with `gitlab_identifier` use the GitLab dataset registry:

```bash
ng_prepare_data "+config_paths=[responses_api_agents/swe_agents/configs/swebench_openhands.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]" \
+output_dirpath=data/swebench_openhands \
+mode=train_preparation \
+should_download=true \
+data_source=gitlab
```

---

## Dataset Registry - GitLab
Expand Down
37 changes: 37 additions & 0 deletions responses_api_agents/swe_agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,43 @@ apt install -y wget && cd /tmp && \
apptainer --version
```

### Download the SWE-bench Verified dataset

The bundled `configs/swebench_openhands.yaml` validation dataset points at NeMo Gym's GitLab/MLflow dataset registry:

```yaml
jsonl_fpath: responses_api_agents/swe_agents/data/swebench_verified_for_sweagent_and_openhands.jsonl
gitlab_identifier:
dataset_name: swebench_verified_for_sweagent_and_openhands
version: 0.0.1
artifact_fpath: swebench_verified_for_sweagent_and_openhands.jsonl
```

Download it with the GitLab dataset command, not the Hugging Face downloader:

```bash
ng_download_dataset_from_gitlab \
+dataset_name=swebench_verified_for_sweagent_and_openhands \
+version=0.0.1 \
+artifact_fpath=swebench_verified_for_sweagent_and_openhands.jsonl \
+output_fpath=responses_api_agents/swe_agents/data/swebench_verified_for_sweagent_and_openhands.jsonl
```

Or let `ng_prepare_data` fetch any missing datasets from the config:

```bash
config_paths="responses_api_agents/swe_agents/configs/swebench_openhands.yaml,\
responses_api_models/vllm_model/configs/vllm_model.yaml"

ng_prepare_data "+config_paths=[$config_paths]" \
+output_dirpath=data/swebench_openhands \
+mode=train_preparation \
+should_download=true \
+data_source=gitlab
```

The GitLab path requires the MLflow/GitLab registry credentials configured for your NeMo Gym environment. If those credentials are unavailable, ask your project admin for registry access or replace the config with a dataset you can access. `ng_download_dataset_from_hf` is only for configs that use `huggingface_identifier`, so it will not resolve this dataset name.

### Step 1 — configure the model

In `env.yaml` at the NeMo-Gym root:
Expand Down