Skip to content

[Refactor] Move the Linux CI jobs to OSDC runners - #4527

Closed
huydhn wants to merge 1 commit into
gh/huydhn/1/basefrom
gh/huydhn/1/head
Closed

huydhn wants to merge 1 commit into
gh/huydhn/1/basefrom
gh/huydhn/1/head

Conversation

@huydhn

@huydhn huydhn commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

Moves the Linux test and lint jobs across 11 workflows onto OSDC pods.

before after
linux.g5.4xlarge.nvidia.gpu mt-l-x86aavx2-11-41-a10g
linux.g6.4xlarge.experimental.nvidia.gpu mt-l-x86aavx2-11-41-l4
linux.12xlarge mt-l-x86iavx512-48-384
linux.4xlarge mt-l-x86iavx512-16-128

Each pod sits on the same instance the label it replaces resolved to, so GPU
model and count are unchanged and only the Kubernetes overhead comes off.
linux_job_v2 -> v3 comes with it, per job: OSDC rejects a job with no
container.

The lint jobs set no runner: at all, so they were quietly taking v2's
linux.2xlarge default. They now pin an OSDC pod explicitly.

On OSDC the checkout runs inside the job container and no stock nvidia/cuda
image ships git, so 42 jobs move to the images test-infra publishes for
this -- the same bases plus git:

nvidia/cuda:12.4.0-devel-ubuntu22.04 -> osdc-cuda:cuda12.4.1-cudnn-devel-ubuntu22.04
nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04 -> osdc-cuda:cuda13.0.3-cudnn-devel-ubuntu24.04
nvidia/cuda:12.8.x-*-ubuntu22.04 -> osdc-cuda:cuda12.8.1-cudnn-devel-ubuntu24.04
nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 -> osdc-cuda:cuda11.8.0-cudnn8-devel-ubuntu22.04
pytorch/pytorch:2.8.0-cuda12.9-cudnn9-devel -> osdc-cuda:cuda12.9.2-cudnn-devel-ubuntu24.04

unittests-robohive keeps nvidia/cudagl:11.4.0-base -- it is the only image
providing the EGL stack that suite renders with -- so it runs standalone
with an explicit git install instead of through v3.

test-linux-libs.yml's unittests-isaaclab stays on EC2: it calls a fork's
reusable workflow that cannot be confirmed to set a container.

docs.yml is handled in #4524. Wheel builds are in #4487.

Authored with Claude Code.

[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4527

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 8, 2026
@github-actions github-actions Bot added Refactoring Refactoring of an existing feature CI Has to do with CI setup (e.g. wheels & builds, tests...) and removed Refactoring Refactoring of an existing feature labels Oct 8, 2026
@github-actions github-actions Bot added the Refactoring Refactoring of an existing feature label Oct 8, 2026
@huydhn

huydhn commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

(Claude, on huydhn's behalf.) Superseded by the gh stack version of the same split: #4530 / #4531 / #4532. Same three commits, same content — this one was built with ghstack by mistake.

@huydhn huydhn closed this Oct 8, 2026
huydhn added a commit that referenced this pull request Oct 8, 2026
Moves the Linux test and lint jobs across 11 workflows onto OSDC pods.

| before | after |
|---|---|
| `linux.g5.4xlarge.nvidia.gpu` | `mt-l-x86aavx2-11-41-a10g` |
| `linux.g6.4xlarge.experimental.nvidia.gpu` | `mt-l-x86aavx2-11-41-l4` |
| `linux.12xlarge` | `mt-l-x86iavx512-48-384` |
| `linux.4xlarge` | `mt-l-x86iavx512-16-128` |

Each pod sits on the same instance the label it replaces resolved to, so GPU
model and count are unchanged and only the Kubernetes overhead comes off.
`linux_job_v2` -> `v3` comes with it, per job: OSDC rejects a job with no
container.

The lint jobs set no `runner:` at all, so they were quietly taking v2's
`linux.2xlarge` default. They now pin an OSDC pod explicitly.

On OSDC the checkout runs inside the job container and no stock nvidia/cuda
image ships git, so 42 jobs move to the images test-infra publishes for
this -- the same bases plus git:

  nvidia/cuda:12.4.0-devel-ubuntu22.04        -> osdc-cuda:cuda12.4.1-cudnn-devel-ubuntu22.04
  nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04  -> osdc-cuda:cuda13.0.3-cudnn-devel-ubuntu24.04
  nvidia/cuda:12.8.x-*-ubuntu22.04            -> osdc-cuda:cuda12.8.1-cudnn-devel-ubuntu24.04
  nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 -> osdc-cuda:cuda11.8.0-cudnn8-devel-ubuntu22.04
  pytorch/pytorch:2.8.0-cuda12.9-cudnn9-devel -> osdc-cuda:cuda12.9.2-cudnn-devel-ubuntu24.04

unittests-robohive keeps nvidia/cudagl:11.4.0-base -- it is the only image
providing the EGL stack that suite renders with -- so it runs standalone
with an explicit git install instead of through v3.

test-linux-libs.yml's `unittests-isaaclab` stays on EC2: it calls a fork's
reusable workflow that cannot be confirmed to set a container.

docs.yml is handled in #4524. Wheel builds are in #4487.

Authored with Claude Code.


ghstack-source-id: b70316e
Pull-Request: #4527
huydhn added a commit that referenced this pull request Oct 8, 2026
Moves the Linux test and lint jobs across 11 workflows onto OSDC pods.

| before | after |
|---|---|
| `linux.g5.4xlarge.nvidia.gpu` | `mt-l-x86aavx2-11-41-a10g` |
| `linux.g6.4xlarge.experimental.nvidia.gpu` | `mt-l-x86aavx2-11-41-l4` |
| `linux.12xlarge` | `mt-l-x86iavx512-48-384` |
| `linux.4xlarge` | `mt-l-x86iavx512-16-128` |

Each pod sits on the same instance the label it replaces resolved to, so GPU
model and count are unchanged and only the Kubernetes overhead comes off.
`linux_job_v2` -> `v3` comes with it, per job: OSDC rejects a job with no
container.

The lint jobs set no `runner:` at all, so they were quietly taking v2's
`linux.2xlarge` default. They now pin an OSDC pod explicitly.

On OSDC the checkout runs inside the job container and no stock nvidia/cuda
image ships git, so 42 jobs move to the images test-infra publishes for
this -- the same bases plus git:

  nvidia/cuda:12.4.0-devel-ubuntu22.04        -> osdc-cuda:cuda12.4.1-cudnn-devel-ubuntu22.04
  nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04  -> osdc-cuda:cuda13.0.3-cudnn-devel-ubuntu24.04
  nvidia/cuda:12.8.x-*-ubuntu22.04            -> osdc-cuda:cuda12.8.1-cudnn-devel-ubuntu24.04
  nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 -> osdc-cuda:cuda11.8.0-cudnn8-devel-ubuntu22.04
  pytorch/pytorch:2.8.0-cuda12.9-cudnn9-devel -> osdc-cuda:cuda12.9.2-cudnn-devel-ubuntu24.04

unittests-robohive keeps nvidia/cudagl:11.4.0-base -- it is the only image
providing the EGL stack that suite renders with -- so it runs standalone
with an explicit git install instead of through v3.

test-linux-libs.yml's `unittests-isaaclab` stays on EC2: it calls a fork's
reusable workflow that cannot be confirmed to set a container.

docs.yml is handled in #4524. Wheel builds are in #4487.

Authored with Claude Code.


ghstack-source-id: b70316e
Pull-Request: #4527
huydhn added a commit that referenced this pull request Oct 9, 2026
Moves the Linux test and lint jobs across 11 workflows onto OSDC pods.

| before | after |
|---|---|
| `linux.g5.4xlarge.nvidia.gpu` | `mt-l-x86aavx2-11-41-a10g` |
| `linux.g6.4xlarge.experimental.nvidia.gpu` | `mt-l-x86aavx2-11-41-l4` |
| `linux.12xlarge` | `mt-l-x86iavx512-48-384` |
| `linux.4xlarge` | `mt-l-x86iavx512-16-128` |

Each pod sits on the same instance the label it replaces resolved to, so GPU
model and count are unchanged and only the Kubernetes overhead comes off.
`linux_job_v2` -> `v3` comes with it, per job: OSDC rejects a job with no
container.

The lint jobs set no `runner:` at all, so they were quietly taking v2's
`linux.2xlarge` default. They now pin an OSDC pod explicitly.

On OSDC the checkout runs inside the job container and no stock nvidia/cuda
image ships git, so 42 jobs move to the images test-infra publishes for
this -- the same bases plus git:

  nvidia/cuda:12.4.0-devel-ubuntu22.04        -> osdc-cuda:cuda12.4.1-cudnn-devel-ubuntu22.04
  nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04  -> osdc-cuda:cuda13.0.3-cudnn-devel-ubuntu24.04
  nvidia/cuda:12.8.x-*-ubuntu22.04            -> osdc-cuda:cuda12.8.1-cudnn-devel-ubuntu24.04
  nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 -> osdc-cuda:cuda11.8.0-cudnn8-devel-ubuntu22.04
  pytorch/pytorch:2.8.0-cuda12.9-cudnn9-devel -> osdc-cuda:cuda12.9.2-cudnn-devel-ubuntu24.04

unittests-robohive keeps nvidia/cudagl:11.4.0-base -- it is the only image
providing the EGL stack that suite renders with -- so it runs standalone
with an explicit git install instead of through v3.

test-linux-libs.yml's `unittests-isaaclab` stays on EC2: it calls a fork's
reusable workflow that cannot be confirmed to set a container.

docs.yml is handled in #4524. Wheel builds are in #4487.

Authored with Claude Code.


ghstack-source-id: b70316e
Pull-Request: #4527
huydhn added a commit that referenced this pull request Oct 9, 2026
Moves the Linux test and lint jobs across 11 workflows onto OSDC pods.

| before | after |
|---|---|
| `linux.g5.4xlarge.nvidia.gpu` | `mt-l-x86aavx2-11-41-a10g` |
| `linux.g6.4xlarge.experimental.nvidia.gpu` | `mt-l-x86aavx2-11-41-l4` |
| `linux.12xlarge` | `mt-l-x86iavx512-48-384` |
| `linux.4xlarge` | `mt-l-x86iavx512-16-128` |

Each pod sits on the same instance the label it replaces resolved to, so GPU
model and count are unchanged and only the Kubernetes overhead comes off.
`linux_job_v2` -> `v3` comes with it, per job: OSDC rejects a job with no
container.

The lint jobs set no `runner:` at all, so they were quietly taking v2's
`linux.2xlarge` default. They now pin an OSDC pod explicitly.

On OSDC the checkout runs inside the job container and no stock nvidia/cuda
image ships git, so 42 jobs move to the images test-infra publishes for
this -- the same bases plus git:

  nvidia/cuda:12.4.0-devel-ubuntu22.04        -> osdc-cuda:cuda12.4.1-cudnn-devel-ubuntu22.04
  nvidia/cuda:13.0.2-cudnn-devel-ubuntu24.04  -> osdc-cuda:cuda13.0.3-cudnn-devel-ubuntu24.04
  nvidia/cuda:12.8.x-*-ubuntu22.04            -> osdc-cuda:cuda12.8.1-cudnn-devel-ubuntu24.04
  nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 -> osdc-cuda:cuda11.8.0-cudnn8-devel-ubuntu22.04
  pytorch/pytorch:2.8.0-cuda12.9-cudnn9-devel -> osdc-cuda:cuda12.9.2-cudnn-devel-ubuntu24.04

unittests-robohive keeps nvidia/cudagl:11.4.0-base -- it is the only image
providing the EGL stack that suite renders with -- so it runs standalone
with an explicit git install instead of through v3.

test-linux-libs.yml's `unittests-isaaclab` stays on EC2: it calls a fork's
reusable workflow that cannot be confirmed to set a container.

docs.yml is handled in #4524. Wheel builds are in #4487.

Authored with Claude Code.


ghstack-source-id: b70316e
Pull-Request: #4527
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Has to do with CI setup (e.g. wheels & builds, tests...) CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Refactoring Refactoring of an existing feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant