Skip to content

feat(blog): poisoned litellm in your CI - #500

Merged
thomasboni merged 9 commits into
mainfrom
blog/litellm-pypi-cicd
Oct 23, 2026
Merged

thomasboni merged 9 commits into
mainfrom
blog/litellm-pypi-cicd

Conversation

@thomasboni

@thomasboni thomasboni commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What

Plumber Incident Brief for the litellm 1.107.0 poisoned PyPI release (October 21, 2026), from the kit in the Notion incident page: src/data/blog/en/litellm-pypi-cicd/index.mdx plus a hero image (SVG source and 1200x675 PNG in the brand style) and an SVG schema of the attack path under "How the attack runs inside a pipeline": the upstream lane (lookalike fork, the project's release workflow, PyPI and the image registries) feeding the five steps inside the reader's pipeline, with the line where pinning breaks the chain.

Structure (per review): intro; What happened as a short sourced summary with the attack path schema; the Plumber take; Are you exposed? with the 60-second check and a detailed check split into your files, GitHub Actions, GitLab CI, machines and images; The fix that would have stopped it (before/after YAML for GitHub Actions and GitLab CI); key takeaways; next steps. Removed on request: the timeline, the step-by-step run inside a pipeline, the weak-link table, the IOC list and the "If you were hit" section.

Review notes:

  • Every shell snippet in the post was run against sample files (requirements.txt, uv.lock, poetry.lock, Pipfile.lock, a workflow, a .gitlab-ci.yml, a pyproject.toml): the version/hash grep hits 1.107.0 in every format and ignores 1.106.2 and other packages; the install-line and unpinned-requirement greps behave as described.
  • pubDate is set to 2026-10-22 (the Notion draft said 2026-10-21); change it if you publish later.
  • About 1,700 words after the readability pass and the section removals, inside the 1,200 to 2,000 target of the incident format.
  • Article URL: https://getplumber.io/blog/litellm-pypi-cicd

Social posts (from the Notion kit, for review in one place)

LinkedIn (Aurel)

litellm 1.107.0 on PyPI shipped files that are not in its git tag.

22.7 million downloads last week. PyPI quarantined the release on October 21.

The risky part is not your laptop. It is the CI job that ran pip install litellm with no version that day, with your cloud role and deploy keys sitting in its environment.

30-second check: grep your workflows for pip install lines that name litellm without ==, then search your October 21 CI logs for litellm-1.107.0.

Pin versions with hashes. Pin images by digest.

Full breakdown + how to check your repos in the first comment.

First comment: Timeline, IOCs, the grep we tested on every lockfile format, and what to rotate: https://getplumber.io/blog/litellm-pypi-cicd?utm_source=linkedin&utm_medium=social&utm_campaign=incident-litellm-pypi-cicd

X thread

1/ litellm 1.107.0 on PyPI shipped 3 files that are not in its git tag. 22.7M downloads last week. PyPI quarantined the release on Oct 21. If your CI installed litellm that day, read on.

2/ How it got in: one line. run: pip install litellm. No version, no hashes. The resolver took the newest release, and Python ran the payload's .pth file on the next start, on the runner, next to your secrets.

3/ Check: search your Oct 21 CI logs for litellm-1.107.0. On machines, run find / -name cook.pth BEFORE running pip or python.

4/ Fix: pin litellm==1.106.2 with hashes, pin the Docker image by digest, rotate what those jobs could read. Full check + tested grep: https://getplumber.io/blog/litellm-pypi-cicd?utm_source=x&utm_medium=social&utm_campaign=incident-litellm-pypi-cicd

Single post / Bluesky: litellm 1.107.0 on PyPI shipped files that are not in its git tag and was quarantined on Oct 21. Any CI job that ran an unpinned pip install litellm that day should rotate its secrets. Check + fix: https://getplumber.io/blog/litellm-pypi-cicd?utm_source=x&utm_medium=social&utm_campaign=incident-litellm-pypi-cicd

Hacker News

Title: Poisoned litellm in your CI
URL: https://getplumber.io/blog/litellm-pypi-cicd

First comment: Author here. We focused on the CI side: the jobs that ran an unpinned pip install litellm on Oct 21 are the ones holding cloud roles and deploy keys. The post has a grep we tested on requirements, uv.lock, poetry.lock and Pipfile.lock, and the order of the cleanup steps matters (don't run pip on a suspect box first, the .pth runs on every Python start). On the root cause: the litellm team now says the GitHub CLI tokens may have been exposed in the logs of an internal deployment reachable from the internet, and that an org owner's token was used to disable branch protection; how the attacker got into that deployment is not confirmed yet. Socket traced the Docker image to the project's release workflow, started with a lookalike source owner, and the litellm team says that run was triggered and approved from two engineers' accounts through what look like stolen GitHub CLI tokens. Happy to be corrected on any of it.

Reddit r/devops

Title: litellm 1.107.0 (PyPI) was quarantined on Oct 21: how to check your CI jobs

Body: If any pipeline ran pip install litellm without a pinned version on Oct 21 from 11:39 UTC, it may have installed 1.107.0, whose wheels carried files that are not in the git tag. Checks: search that day's CI logs for litellm-1.107.0; on self-hosted runners run find / -name cook.pth -o -name cook.py -o -name litellm-1.107.0.dist-info 2>/dev/null before running pip or python. If you get a hit, rotate every secret the job could read, clear dependency caches, and pin litellm==1.106.2 with hashes. Also check images on litellm/litellm or ghcr.io/berriai/litellm* (incl. -database, -non_root) tags main-latest, main-stable, latest or v1.107.0. Write-up with IOCs and a tested grep: https://getplumber.io/blog/litellm-pypi-cicd?utm_source=reddit&utm_medium=social&utm_campaign=incident-litellm-pypi-cicd

Reddit r/github

Title: Checking GitHub Actions runs for the litellm 1.107.0 PyPI release

Body: gh run list --created '>=2026-10-21' --json databaseId,workflowName,createdAt then gh run view <id> --log | grep litellm-1.107.0 tells you whether a workflow installed the quarantined release. Also delete Actions caches built that day (gh cache list / gh cache delete <id>): a venv cache with an unchanged key can bring it back. Details: https://getplumber.io/blog/litellm-pypi-cicd?utm_source=reddit&utm_medium=social&utm_campaign=incident-litellm-pypi-cicd

🤖 Generated with Claude Code

https://claude.ai/code/session_01MnHoxB75RcarpRdtUfCC7U

Plumber Incident Brief for the poisoned litellm 1.107.0 PyPI release of
October 21, 2026: exposure check, sourced timeline, attack path in the
pipeline, the controls that cover the container path, full check with IOCs,
response plan, before/after YAML for GitHub Actions and GitLab CI, and the
Plumber take. Hero image as SVG source plus the rendered PNG.
A fenced block ends a markdown list, so steps 2 and 3 of the aside collapsed
into one paragraph; the three steps are bold paragraphs now.
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Preview deployment for 1d2534d: https://getplumber-gegarycdh-getplumber.vercel.app

An SVG schema under the attack-path section: the upstream lane (lookalike
fork, the project's release workflow, PyPI and the image registries) feeding
the five steps inside the reader's pipeline, and the line where pinning
breaks the chain.
…s questions

Intro, then What happened as a short sourced summary with the schema, the
Plumber take, Are you exposed? with the 60-second check and a detailed check
split into your files, GitHub Actions, GitLab CI, machines and IOCs, then the
detailed description of the attack (timeline, how it ran, weak link, the fix).
The If you were hit section is removed; cache and mirror cleanup moved into
the provider checks.
@thomasboni thomasboni changed the title feat(blog): litellm 1.107.0 incident brief, the poisoned PyPI release and your CI feat(blog): what poisoned litellm release means for your CI Oct 22, 2026
Short paragraphs, bullet lists for the release facts, the publish path, what
was reachable, the checks and the IOCs, and backticks on every version, file,
tag and tool. Facts, sources and the tested shell snippets are unchanged.
@thomasboni thomasboni changed the title feat(blog): what poisoned litellm release means for your CI feat(blog): poisoned litellm in your CI Oct 23, 2026
…itellm brief

The timeline, the step-by-step run inside a pipeline, the weak-link table and
the IOC list are removed; the before/after YAML fix becomes its own section
after the checks.
@thomasboni
thomasboni merged commit 4398ae7 into main Oct 23, 2026
9 checks passed
@thomasboni
thomasboni deleted the blog/litellm-pypi-cicd branch October 23, 2026 00:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant