Notes on coding agents and the pages they leave behind, on moving between notes apps, and on the plain mechanics of keeping what you think. Written and published from a Sumibako vault.
The Railway breach: what encrypted environment variables meant, and how to rotate after it
Railway, Zeabur and TiDB Cloud lost customers' secrets in one week. What encrypted meant, how Zeabur's second breach grew from its first, and what to rotate.
Prashant Shaw
Founder, Sumibako
10 min read
Every app has a drawer where the keys live. On a hosting platform it is called environment variables, and into it go the database password, the Stripe key, the OpenAI key, the GitHub token, the credentials for the private registry. You do not think about the drawer. That is the point of it. Like every drawer of keys, it also holds two or three that nobody can identify and nobody dares throw away.
In the week of 20 October, three platforms that hold such drawers for other people reported that someone had been in them. Railway said an intruder reached encrypted environment variables and decrypted some. Zeabur said its platform database was read, 3.2 million variables and all, and that it is shutting down. PingCAP said a stolen service account had been inside TiDB Cloud's internal systems for a month. Sumibako's own drawer sits in Vercel and Convex, which is the same shape, so I read all three reports closely. This is what they say, in their own words, and what to do if any of those drawers was yours.
🚂 What Railway says happened
Railway's first disclosure, by Jacob Cooper on 21 October, opens plainly: "Earlier this morning, Railway detected anomalous activity involving systems that store customer environment variables." The finding: "an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents."
The preliminary report a day later gives the window, "from October 20th 13:58 UTC to October 22nd 03:32 UTC", and the motive: "Early investigations tell us this attack was attempting to obtain AI provider API keys to perform distillation attacks." Distillation is training your own model on another model's answers, and it costs a fortune in API calls unless the API calls are on somebody else's key. Railway's description of the attacker is the most 2026 sentence of the week: "Initial evidence leads us to believe that the malicious actor had access to abliterated models and substantial compute resources which enabled them to perform highly sophisticated exploits against our infrastructure."
How they got in, Railway is keeping until its final report. No CVE has been assigned.
🔐 What "encrypted" bought: about three hours
The word doing the most work in every headline is encrypted, so it is worth reading what Railway's own timeline says it meant. Step one, 04:24 UTC on the 21st: "Continued forensic research identified the attacker was able to pull secrets, in encrypted format; however, at the time attacker was unable to decrypt said secrets." Step two, from 05:27: "At this point, the attacker accessed credentials to access our CI environment where the attacker was able to decrypt artifacts that held additional API access." Step three, from 07:25: "Railway engineering identified database evidence of an unauthorized query of encrypted environment variables, of which a subset were decrypted utilizing a combination of the compromised credentials."
Railway's own timeline. The shaded rows are the time encryption at rest bought.
So encryption at rest did what it does. It turned a stolen database into a locked box, and the box stayed locked for roughly three hours, until the intruder found the keys to it in the CI system. Encryption is a delay, and a delay is worth having, but it is not the same as the secrets being safe.
It also has a consequence for anyone trying to find out whether they were hit. "Because the full keys were envelope encrypted, this query alone cannot identify specifically which keys in the affected customers project/service were affected." Railway cannot tell you which of your variables were read. GitGuardian's advice follows from that: "Treat every secret stored in Railway as at risk until investigated." Including the ones Railway calls sealed, which the dashboard cannot show you: "That limits casual exposure, but the disclosure does not say whether they were in scope. Rotate them too."
📏 1.85%, about 100,000 workspaces, and more than that
Railway's figure for the damage: "we can validate that at most 1.85% of all workspaces had some environment variables potentially compromised. This represents, numerically, about 100,000 workspaces." The number of people who felt it was larger, because Railway asked the providers whose keys it held to revoke them, and it could not be selective: "As such, the amount of users who were affected by upstream provider reset requests was larger than 100k." If your OpenAI or Anthropic or Stripe key stopped working on 21 October, Railway asks you to read that correctly: "Users being notified by vendors about revocation does not imply that their credentials were impacted by the incident."
Railway also explains why its customers heard about it hours after it knew: "As the incident was on-going, we wanted to ensure the attacker was ejected from our running infrastructure before telling users to roll their credentials, to prevent the credentials from becoming immediately compromised again." That is a defensible choice and a hard one, and it is the kind of thing worth knowing about a platform before the day you need it.
Then the week's most human mistake. On 23 October the AI providers asked for another batch of possibly exposed keys, and Railway sent it, "but because we could not filter keys by when they were added, it inadvertently included newer keys", so some people had the replacement keys they had just created revoked as well. By 24 October the status page had a line nobody writes in a good week: "Railway is not currently under attack." And for anyone outside the groups Railway emailed directly: "Workspaces outside those higher-risk groups are not confirmed safe from compromise."
🧯 Zeabur: the second breach was the first one's leftovers
Zeabur, a smaller platform of the same kind, had been breached in August. Its October report describes the second one: "an attacker used leaked Zeabur internal credentials to break into the shared cluster, some Dedicated Servers, and Zeabur internal systems, and eventually read the platform database." The root cause, in a sentence I wish more companies could write: "The root cause of this incident lies with Zeabur: long-lived admin credentials were stored in plaintext and could be used from the public internet."
Where the credentials came from is the lesson. "The admin tokens had no expiry, had not been rotated since they were created in January 2024, and were not included in the post-August rotation." Zeabur's August report had said all internal admin keys were rotated. The October one corrects it: "That statement was incomplete, and we are correcting it here." A rotation is only as complete as the inventory it works from, and the keys nobody listed were the keys that let the attacker back in.
The numbers, via BleepingComputer: the attacker was able to "steal over 3.2 million environment variable records and information for 321,810 accounts." Detection: "More than 16 hours of attack activity on October 14 triggered no alerts; the team identified the incident the next day after a user report." The motive matches Railway's: "the attacker was mainly looking for API keys they could use directly, especially keys for AI services". And one key had already been spent: "From October 16, 01:50 to 02:56, an OpenAI API key that a user stored on Zeabur was used to send about 1,107 image generation requests."
Zeabur is closing on 31 December. Its explanation, quoted by BleepingComputer: "Following this latest incident, we reassessed what it would take to keep operating and the responsibilities our team could carry. We ultimately decided to wind down." That is the other thing a drawer on someone else's platform can do: disappear, with ten weeks' notice.
🗄️ TiDB Cloud: a month inside
PingCAP's report on its TiDB Cloud database service is the quietest of the three and the longest in duration: "Between September 9 and October 7, 2026, an unauthorized party used stolen service account credentials to access certain PingCAP internal systems, copy data, and obtain source code. PingCAP became aware of the incident on October 7 and began disabling the affected credentials the same day."
What left: "Exported records included user names and email addresses, organization and project associations, and SQL editor text and errors." Also billing records and contracts. On the keys: "An exported table contained encrypted TiDB Cloud API-key values. One investigator-owned value tested by the external firm could not be used directly as an API key." And the line customers most wanted: "there is no evidence that data stored in TiDB was impacted." The action is to replace the management API keys anyway.
🔗 One campaign or three?
Railway
Zeabur
TiDB Cloud (PingCAP)
What was read
Encrypted environment variables, a subset decrypted
The platform database: 3,218,965 variable records, 321,810 accounts
Internal systems: account, SQL editor and billing records, encrypted API key values, source code
When
20 Oct 13:58 to 22 Oct 03:32 UTC
14 to 15 Oct
9 Sep to 7 Oct
The way in
Not yet published
Plaintext admin tokens from January 2024, missed by the August rotation
Stolen service account credentials
Motive, in their words
AI provider API keys, for distillation
API keys for AI services
Not stated
Customers told to
Roll every variable and registry credential
Replace every credential, again, without waiting for an email
Replace management API keys
Status on 24 Oct
Final report pending; new signups paused
Shutting down on 31 December 2026
Credentials disabled on 7 Oct; investigation continues
Railway thinks it is part of something: "We believe this is part of a series of attacks recently targeting platform companies with the aim of compromising AI inference providers." Zeabur blames its own leftover credentials and suspects they came out of its August breach. PingCAP mentions no AI motive at all. Rescana, which grouped the three, is careful: "No common root cause or shared upstream provider has been confirmed by any primary source." Two of the three attackers wanted AI keys; that is a pattern in what thieves want, not proof of one thief.
🔄 Rotate at the issuer: the one instruction all three agree on
Railway's instruction is the template: "we strongly recommend that you immediately roll all environment variables, including any API keys, stored in Railway, as well as any private registry credentials, and review logs for those services for unusual activity." It published thirteen guides, one per provider, from checking OpenRouter usage to reconnecting GitHub. Zeabur's version has the urgency of a second time round: "Please act now; do not wait for a notification email. Credentials you replaced after the August incident must be replaced again."
Three details matter more than the list.
1️⃣ Rotate where the key was issued, not where it was stored. Rescana puts it in two sentences: "Rotate at the issuer, not just on the platform. Changing a variable on the platform does not revoke the old key." Zeabur says the same about databases: "Changing environment variables on Zeabur does not change the passwords inside the database."
2️⃣ Then read the provider's usage page for the window. Rotation stops future use. Only the logs tell you about the past, and the Zeabur user with 1,107 image generations on their bill found out that way.
3️⃣ Rotate from a clean place and keep a list. Railway's own slip on 23 October, revoking the replacements, is what happens without an inventory of which key is which and when it was made. A plain text file is enough, one line per key:
📝NOTE
Sumibako's position, since I am asking you to think about yours: the app's secrets live in Vercel's and Convex's environment settings, the same shape as Railway's. What limits the damage if one of them is read is not the platform. It is that each key does one thing, that each can be replaced in minutes, and that the tokens Sumibako issues to agents are stored only as hashes, so a copy of the database holds nothing that logs in.
The rest of the week, from PHP to a certificate authority whose private key was the number 4, is in the state of the internet this month. The habit of treating every secret as lent, with the rest of what this week taught, is in a checklist for the next one. Sumibako's own drawer is next on my list. Writing two thousand words about other people's keys turns out to be an effective way to remember that you have some.
Eight eMudhra keys were integers under 256, one of them 4. Why browsers blocked them in hours, why your Docker image did not, and what to check by 27 October.
10 min read
provider purpose scope created rotated usage checked
<who issued> <what it does> <one service> <date> <date> <yes, up to when>
