Repository navigation
feat(blog): poisoned litellm in your CI - #500
Merged
Merged
Conversation
Plumber Incident Brief for the poisoned litellm 1.107.0 PyPI release of October 21, 2026: exposure check, sourced timeline, attack path in the pipeline, the controls that cover the container path, full check with IOCs, response plan, before/after YAML for GitHub Actions and GitLab CI, and the Plumber take. Hero image as SVG source plus the rendered PNG.
A fenced block ends a markdown list, so steps 2 and 3 of the aside collapsed into one paragraph; the three steps are bold paragraphs now.
|
Preview deployment for 1d2534d: https://getplumber-gegarycdh-getplumber.vercel.app |
An SVG schema under the attack-path section: the upstream lane (lookalike fork, the project's release workflow, PyPI and the image registries) feeding the five steps inside the reader's pipeline, and the line where pinning breaks the chain.
…s questions Intro, then What happened as a short sourced summary with the schema, the Plumber take, Are you exposed? with the 60-second check and a detailed check split into your files, GitHub Actions, GitLab CI, machines and IOCs, then the detailed description of the attack (timeline, how it ran, weak link, the fix). The If you were hit section is removed; cache and mirror cleanup moved into the provider checks.
Short paragraphs, bullet lists for the release facts, the publish path, what was reachable, the checks and the IOCs, and backticks on every version, file, tag and tool. Facts, sources and the tested shell snippets are unchanged.
…itellm brief The timeline, the step-by-step run inside a pipeline, the weak-link table and the IOC list are removed; the before/after YAML fix becomes its own section after the checks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Plumber Incident Brief for the litellm 1.107.0 poisoned PyPI release (October 21, 2026), from the kit in the Notion incident page:
src/data/blog/en/litellm-pypi-cicd/index.mdxplus a hero image (SVG source and 1200x675 PNG in the brand style) and an SVG schema of the attack path under "How the attack runs inside a pipeline": the upstream lane (lookalike fork, the project's release workflow, PyPI and the image registries) feeding the five steps inside the reader's pipeline, with the line where pinning breaks the chain.Structure (per review): intro; What happened as a short sourced summary with the attack path schema; the Plumber take; Are you exposed? with the 60-second check and a detailed check split into your files, GitHub Actions, GitLab CI, machines and images; The fix that would have stopped it (before/after YAML for GitHub Actions and GitLab CI); key takeaways; next steps. Removed on request: the timeline, the step-by-step run inside a pipeline, the weak-link table, the IOC list and the "If you were hit" section.
Review notes:
.gitlab-ci.yml, a pyproject.toml): the version/hash grep hits 1.107.0 in every format and ignores 1.106.2 and other packages; the install-line and unpinned-requirement greps behave as described.pubDateis set to 2026-10-22 (the Notion draft said 2026-10-21); change it if you publish later.Social posts (from the Notion kit, for review in one place)
LinkedIn (Aurel)
litellm 1.107.0 on PyPI shipped files that are not in its git tag.
22.7 million downloads last week. PyPI quarantined the release on October 21.
The risky part is not your laptop. It is the CI job that ran
pip install litellmwith no version that day, with your cloud role and deploy keys sitting in its environment.30-second check: grep your workflows for
pip installlines that name litellm without==, then search your October 21 CI logs forlitellm-1.107.0.Pin versions with hashes. Pin images by digest.
Full breakdown + how to check your repos in the first comment.
First comment: Timeline, IOCs, the grep we tested on every lockfile format, and what to rotate: https://getplumber.io/blog/litellm-pypi-cicd?utm_source=linkedin&utm_medium=social&utm_campaign=incident-litellm-pypi-cicd
X thread
1/ litellm 1.107.0 on PyPI shipped 3 files that are not in its git tag. 22.7M downloads last week. PyPI quarantined the release on Oct 21. If your CI installed litellm that day, read on.
2/ How it got in: one line.
run: pip install litellm. No version, no hashes. The resolver took the newest release, and Python ran the payload's .pth file on the next start, on the runner, next to your secrets.3/ Check: search your Oct 21 CI logs for
litellm-1.107.0. On machines, runfind / -name cook.pthBEFORE running pip or python.4/ Fix: pin litellm==1.106.2 with hashes, pin the Docker image by digest, rotate what those jobs could read. Full check + tested grep: https://getplumber.io/blog/litellm-pypi-cicd?utm_source=x&utm_medium=social&utm_campaign=incident-litellm-pypi-cicd
Single post / Bluesky: litellm 1.107.0 on PyPI shipped files that are not in its git tag and was quarantined on Oct 21. Any CI job that ran an unpinned
pip install litellmthat day should rotate its secrets. Check + fix: https://getplumber.io/blog/litellm-pypi-cicd?utm_source=x&utm_medium=social&utm_campaign=incident-litellm-pypi-cicdHacker News
Title: Poisoned litellm in your CI
URL: https://getplumber.io/blog/litellm-pypi-cicd
First comment: Author here. We focused on the CI side: the jobs that ran an unpinned
pip install litellmon Oct 21 are the ones holding cloud roles and deploy keys. The post has a grep we tested on requirements, uv.lock, poetry.lock and Pipfile.lock, and the order of the cleanup steps matters (don't run pip on a suspect box first, the .pth runs on every Python start). On the root cause: the litellm team now says the GitHub CLI tokens may have been exposed in the logs of an internal deployment reachable from the internet, and that an org owner's token was used to disable branch protection; how the attacker got into that deployment is not confirmed yet. Socket traced the Docker image to the project's release workflow, started with a lookalike source owner, and the litellm team says that run was triggered and approved from two engineers' accounts through what look like stolen GitHub CLI tokens. Happy to be corrected on any of it.Reddit r/devops
Title: litellm 1.107.0 (PyPI) was quarantined on Oct 21: how to check your CI jobs
Body: If any pipeline ran
pip install litellmwithout a pinned version on Oct 21 from 11:39 UTC, it may have installed 1.107.0, whose wheels carried files that are not in the git tag. Checks: search that day's CI logs forlitellm-1.107.0; on self-hosted runners runfind / -name cook.pth -o -name cook.py -o -name litellm-1.107.0.dist-info 2>/dev/nullbefore running pip or python. If you get a hit, rotate every secret the job could read, clear dependency caches, and pinlitellm==1.106.2with hashes. Also check images onlitellm/litellmorghcr.io/berriai/litellm*(incl.-database,-non_root) tagsmain-latest,main-stable,latestorv1.107.0. Write-up with IOCs and a tested grep: https://getplumber.io/blog/litellm-pypi-cicd?utm_source=reddit&utm_medium=social&utm_campaign=incident-litellm-pypi-cicdReddit r/github
Title: Checking GitHub Actions runs for the litellm 1.107.0 PyPI release
Body:
gh run list --created '>=2026-10-21' --json databaseId,workflowName,createdAtthengh run view <id> --log | grep litellm-1.107.0tells you whether a workflow installed the quarantined release. Also delete Actions caches built that day (gh cache list/gh cache delete <id>): a venv cache with an unchanged key can bring it back. Details: https://getplumber.io/blog/litellm-pypi-cicd?utm_source=reddit&utm_medium=social&utm_campaign=incident-litellm-pypi-cicd🤖 Generated with Claude Code
https://claude.ai/code/session_01MnHoxB75RcarpRdtUfCC7U