Skip to content

docs: unclear how to obtain SWE-bench dataset #1417

Description

@cwing-nvidia

A user trying to run SWE-bench Verified rollouts was blocked on obtaining the dataset. The YAML config (swebench_openhands.yaml) references gitlab_identifier with dataset_name: swebench_verified_for_sweagent_and_openhands, but responses_api_agents/swe_agents/README.md doesn't explain how to download it. The user tried ng_download_dataset_from_hf and got a 401.

It's unclear from the docs:

  • Where the dataset is hosted and how to obtain it
  • Which download command to use (ng_download_dataset_from_hf, ng_download_dataset_from_gitlab, ng_prepare_data, etc.)
  • What credentials (if any) are needed

Additional issues in responses_api_agents/swe_agents/README.md:

  • Example commands use internal Lustre paths (e.g. /lustre/fsw/portfolios/llmservice/users/...) that are not useful to external users
  • SIF image creation is documented only as a link to an external script in the NeMo-Skills repo (nemo_skills/dataset/swe-bench/dump_images.py). The SIF creation workflow should be self-contained within NeMo Gym's own docs — users should not need to go to a separate repo to figure out how to set up container images.

Activity

  1. added
    community-requestIssue reported or requested by someone from the community
    documentationImprovements to documentation
    and removed
    community-requestIssue reported or requested by someone from the community
    on May 26, 2026
  2. added theissue type on May 26, 2026
  3. changed the title [-]docs: unclear how to obtain SWE-bench dataset for swe_agents[/-] [+]docs: unclear how to obtain SWE-bench dataset[/+] on May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions