Notes on coding agents and the pages they leave behind, on moving between notes apps, and on the plain mechanics of keeping what you think. Written and published from a Sumibako vault.
LiteLLM 1.107.0 and the OVHcloud CLI: two stolen tokens, two poisoned downloads, one pinned hash
A key committed in May became a credential stealer on PyPI in October; five OVHcloud CLI releases were swapped for 94 minutes. What to do if you got either.
Prashant Shaw
Founder, Sumibako
10 min read
If you ran pip install litellm between 11:39 and 13:23 UTC on 21 October, you installed a credential stealer from the official package index, built by the project's own release pipeline, with valid provenance. Two days earlier, if you downloaded the OVHcloud command line tool from its official GitHub releases page during a 94-minute window, you got a different stealer, uploaded by a legitimate contributor's account. Neither attack involved a bug in the software. Both involved a human's token.
I publish a small CLI of my own and I pulled both post-mortems apart to see what would have saved me. The honest answer is: a hash, a short-lived token, and a setting that was off. The dishonest answer, which I prefer, is that nobody would bother attacking a tool with my download numbers. OVHcloud's contributor could have said the same in March, which is when his account was taken. Here is what happened, in the words of the people who were there.
📦 Two hours on PyPI
LiteLLM is a Python proxy that lets an application talk to any model provider through one interface. Datadog's analysis calls it "a popular Python proxy library for LLM providers with about 23 million weekly downloads." On 21 October it had a bad release. LiteLLM's incident post: "A malicious litellm==1.107.0 package was published to PyPI and was live for approximately 2 hours (11:39 UTC to 13:23 UTC). It has since been pulled." The Docker tags, including latest, "were compromised from 11:46 UTC to 14:20 UTC".
The code in GitHub was clean. "The official v1.107.0 source tag in GitHub does not contain the malicious files. The compromised artifacts were built from an attacker-controlled repository." Which is why nobody reading the repository would have seen it coming, and why StepSecurity's
points at the real exposure: "many projects depend on it with an open ended litellm>= constraint, so a fresh install or lockfile refresh would have picked up 1.107.0 as soon as it was published."
How long each poisoned download was live, and when someone first said so in public.
🐍 What the package did when Python started
The payload did not wait to be imported. Datadog: "Python runs .pth files and sitecustomize.py automatically every time the interpreter starts. Any Python process in the environment runs the malware, even if LiteLLM is never imported". It then collected, per the first report, "the environment variables of every readable process" and, where a LiteLLM proxy had a database, "every upstream provider key the proxy stores", encrypted the bundle, and posted it to a lookalike domain that, StepSecurity found, "was registered on October 13, 2026 at 00:45 UTC". The attack was staged eight days before it ran.
There was a second stage, a loader that would fetch more code every hour from a dead-drop GitHub account. It never fired: "That file only ever held test values, so no second stage payload was observed in the wild." It is not dead, though. "The channel is tied to the username, so anyone who later registers letters9813 and recreates pythonclient-test/main/domain.txt would get code execution on every host that is still infected." A machine that installed 1.107.0 and was merely cleaned of the package is a machine waiting for whoever registers a deleted username.
🔑 How they got in: a key from May and a token past 2FA
LiteLLM's root cause post, published on 23 October, corrects its own first account and is unusually complete. It starts in May, when a configuration file holding the master key of an internal service was committed to a public repository. "However, key revocation did not occur for the master key." Five months later: "On October 10th, the threat actor used this key to gain access to our LiteLLM Agent Platform and extracted this GitHub CLI OAuth token. From this point onwards, the threat actor had owner-level access to all repositories in our GitHub."
Two-factor authentication was on. It did not matter, because a GitHub CLI token is issued after the second factor and is not asked for it again. The token, LiteLLM says, "was not associated with a service account and was not scoped down." Two more maintainers' tokens were lifted through throwaway CI workflows whose run logs were then deleted.
Then a coincidence of timing. The release gate had been loosened days earlier: "This restriction was relaxed earlier this week to accommodate the temporary push for more frequent releases and bugfix backports, allowing any two engineers to trigger and approve a release." Two stolen tokens were two engineers. On the morning of the 21st the owner's token overrode branch protection, pointed the release workflow at a lookalike fork, and the pipeline did its job. StepSecurity's summary of why the hardening LiteLLM added after a compromise in March did not help: "None of that helped here, because the attacker did not bypass the pipeline. With an organization owner's token, the attacker changed the trusted release workflow itself, so the malicious artifacts were built, signed, and published by the legitimate pipeline with valid provenance."
LiteLLM's first post had blamed a different mechanism. The second says so: "Our first post suspected the tokens were taken from the logs of an internal LiteLLM deployment. The request logs show the first token came from a different service, described in step 1." I note this because the correction is the most useful thing in the whole story: a company that publishes its wrong theory and then its right one is a company you can learn from.
Datadog's Ian Kretz draws the line for maintainers: "Maintainers should lock release workflows to approved repositories and treat developer CLI tokens as the high-value secrets that they are."
When (UTC)
What happened
27 May 18:53
A config file holding an internal service's master key is committed to a public repository. It is never revoked
10 Oct 21:27
The key opens the Agent Platform API, which returns an organisation owner's GitHub CLI token
10 Oct 22:11
Throwaway CI workflows post two more maintainers' tokens to a webhook; the runs are deleted
13 Oct 00:45
The lookalike domain litellm.dev is registered; the dead-drop GitHub account six hours later
21 Oct 10:08
Branch protection rejects a change to the release workflow; five seconds later the owner's token overrides it
21 Oct 11:11
A second stolen token approves the production release gate
21 Oct 11:39
Seven malicious wheels reach PyPI; Docker Hub and GHCR images follow at 11:46
21 Oct 12:06
Upwind's researcher opens issue #47826
21 Oct 13:23
PyPI release pulled; GHCR tags gone by 13:51, Docker Hub by 14:20
23 Oct 21:00
LiteLLM's attack chain post corrects its first account of the breach
⏱️ Caught in twenty minutes
The release was found by a researcher reading the wheel's file manifest, not by any automated defence. The report on GitHub: "Reported by the Upwind security research team. We recovered the full payload by static decoding only; nothing was executed." LiteLLM credits them with "catching the malicious release within 20 minutes of publication". The reporter's closing request is the one every Python maintainer should read twice: "Consider moving to PyPI Trusted Publishing with attestations. litellm has never published attestations, so users currently can't tell a tampered build from a legitimate one."
🇫🇷 OVHcloud: 94 minutes and a laptop build
Two days before, a different shape of the same thing. Octave Klaba, OVHcloud's founder, announced it on X in French: "La version CLI de 14h47 a été compromise par un hacker. Nous l'avons détecté et bloqué à 16h21." The affected versions were v0.11.0 through v0.15.0, five releases published between April and September, whose downloadable files were replaced in place. The Hacker News has the window: "They were live on GitHub for up to 94 minutes, from 12:47 to 14:21 UTC, and reached only people who installed or updated the OVHcloud CLI during that window, via a path that checks for no hash, on macOS, Linux x86_64 or arm64, or Windows x86_64."
No pipeline was involved. "The source code, the release tag, and the company's build pipeline were not touched: the bad build was a plain go build of a later commit, uploaded straight onto releases published between April and September." The upload came from a contributor's account, which OpenSourceMalware's forensic timeline describes as "a legitimate OVH contributor, using the ordinary GitHub Releases API. No tag was pushed and no GitHub Actions workflow ran." And: "The HassenBF account was already compromised and had been for seven months."
Who noticed? "A customer's CI job caught the swap because it pins the SHA-256 of the Linux arm64 archive and began failing." One customer, one pinned hash, one failing build, and the files were gone 31 minutes after the issue was filed. OpenSourceMalware's one-line verdict: "The one person who noticed had hand-pinned a SHA-256 in their own CI."
Klaba's second post named the setting: "Le problème vient de l'option l’immutabilité des releases sur Github qui n'a pas été activée." GitHub can lock a release's files once published; OVHcloud had not turned it on. It has now: "L'option a été activée. Fin de l'incident." With the caveat The Hacker News adds, that immutability "locks a release's files and tag but protects only releases created after it is enabled." What the swapped binary stole, nobody can say: "OVHcloud has not said which files the binary reads, where it sends them, or how many copies were downloaded, and no copy of a swapped archive has been analyzed in public." Klaba's advice, as BleepingComputer renders it: "The quality of the binary is poor but it can go all the way. If you are affected, please change all the secrets on your disk."
🧾 If you installed either
⚠️WARNING
LiteLLM's instruction is about the machine, not the package: "If an environment started Python with litellm==1.107.0, treat that environment as compromised." And do not reach for pip first: "Any Python start in the affected environment runs cook.pth and sitecustomize.py again, and that includes running pip". "If 1.107.0 was ever installed on a host with outbound internet, assume the exfil succeeded."
1️⃣ From a different, clean machine, rotate everything the affected one could read: cloud keys, provider keys, database passwords, the keys a LiteLLM proxy stored for its upstream providers. OpenSourceMalware's order for the OVHcloud case is the same: "Rotate from a different machine, then remove the binary and install v0.16.0 or later." The Hacker News explains why that order: "Updating replaces the program but does not revoke a key that was already copied, which is why the rotation comes first."
2️⃣ Rebuild rather than clean. LiteLLM: "Rebuilding images and venvs from scratch is better." Then pin: "For the time being, please pin your version to 1.106.2 or any explicit version published prior to now."
3️⃣ If you pulled a LiteLLM Docker image on latest or main-latest between 11:46 and 14:20 UTC on 21 October, the same applies to the container host.
🧷 The switches that were off
Three switches, three stories. OVHcloud's release immutability was off, so published files could be replaced. LiteLLM published through a trusted pipeline, but the pipeline's own workflow could be edited by any owner's token, and it never published attestations that would let a user compare a wheel to a build. And the tokens in both cases were long-lived and broad.
None of this is new. After the Shai-Hulud worm of 2025, which GitHub described as "a self-replicating worm that infiltrated the npm ecosystem via compromised maintainer accounts by injecting malicious post-install scripts into popular JavaScript packages.", npm moved to short-lived tokens and trusted publishing, which GitHub notes "was pioneered by PyPI in April 2023 as a way to get API tokens out of build pipelines." And in March 2024 the xz backdoor taught everyone that the download is not the repository: "The upstream xz repository and the xz tarballs have been backdoored."
OpenSourceMalware's sentence is the one to remember when you set up a project's releases: "Every maintainer’s personal account is a release-signing key. Compromise one and you can publish." LiteLLM's own last word is the other: "This incident has provided a strong illustration of the risks associated with failing to rotate a potentially exposed key."
For my own CLI this is the week's homework, in three lines: release immutability on, a publish token minted per run that dies with it, and a hash in the install instructions so that someone, somewhere, has a CI job that fails when it should. The wider week is in the state of the internet this month, and the switches above are three lines of a checklist for the next one.
Eight eMudhra keys were integers under 256, one of them 4. Why browsers blocked them in hours, why your Docker image did not, and what to check by 27 October.
10 min read
shellscript
# Was 1.107.0 ever here? Its dropped files run at every Python start.find / -name cook.pth -o -name cook.py 2>/dev/null# requirements.txt: an exact version, not a floorlitellm==1.106.2# What caught the OVHcloud swap: a checksum pinned in CI, checked before useecho '<expected sha256> <downloaded archive>' | sha256sum -c -
