← All notes

Field note

Which env vars to rotate after the Railway breach

Railway's October 2026 disclosure says some decrypted environment variables may have left the platform. What to roll, what Railway's Postgres guide actually says, and what a publishable key is not.

23 Oct 20267 min read1501 words

On 21 October 2026 Railway published a security disclosure. The short version, in their words: an unauthorized third party may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents. Those variables may contain API keys or database and registry credentials. Railway said the impact might be limited to a subset of users and still told every user to roll sensitive variables stored on the platform.

This note is the rotation list I want in front of a vibe-coded app. It follows Railway's disclosure, their preliminary report, and their PostgreSQL guide. It does not claim your project was in the affected set. A host incident and a key pasted into client JavaScript are different failures. Both end with the same action at the provider that issued the secret: revoke the old value.

The rotation checklist is the same list, shorter, on the site. The free scan reads the public JavaScript. It does not read Railway's environment.

What Railway said happened

The disclosure is dated Wednesday 21 October 2026. Railway detected anomalous activity on systems that store customer environment variables. The investigation, as they described it that morning, indicated that an unauthorized party may have accessed encrypted variables and decrypted some of them. They said they believed the activity was part of a series of attacks aimed at AI inference providers, and that they had notified those providers to revoke keys.

Their recommendation was not "wait until we email you." They wrote that every user should immediately roll all environment variables, including any API keys, stored in Railway, plus any private registry credentials, and review logs for those services. Deployments were paused while they investigated. A later email would go to accounts they confirmed were affected. The precaution applied before that email.

The preliminary report adds a timeline and one limit you should not skip. On 21 October, Railway identified a database query of encrypted environment variables, of which a subset were decrypted using compromised credentials. Because the values were envelope encrypted, that query alone cannot name which keys in an affected project were decrypted. Logs from the attacker's payloads showed the API key names they were searching for. Railway notified vendors and, for some services, sent the keys they had registered on the platform so those vendors could revoke them.

Two consequences follow. You cannot look at the preliminary report and decide your OPENAI_API_KEY was spared because the query could not list it. And a vendor revoke, where Railway asked for one, does not cover a secret the vendor does not know about: a database password, a webhook signing secret, a Neon connection string you pasted by hand.

Roll the Railway variables, then the copies

Open each service's variables. Anything that grants access gets a new value at the issuer, then the new value goes back into Railway, then the service is redeployed. Containers read variables at startup. A rotated dashboard value does nothing for a process that is still holding the old one.

Treat these as secrets if they were stored on Railway:

  • Database URLs and passwords, including DATABASE_URL, DATABASE_PUBLIC_URL, and POSTGRES_PASSWORD.
  • Supabase secret or legacy service_role keys. The publishable anon key is a different object, covered below.
  • Stripe secret keys and webhook signing secrets (whsec_). The publishable pk_ key is not this item.
  • OpenAI keys (sk-, sk-proj-, sk-admin-, sk-svcacct-).
  • Clerk's CLERK_SECRET_KEY. That value is sk_test_ or sk_live_, the same prefixes Stripe uses. The publishable pk_ key is not this item.
  • Neon connection strings. The password in the URL is the credential.
  • Private registry credentials Railway called out by name.

A reference variable such as a pointer at another service's DATABASE_URL updates when that other service changes, on the next deploy. A string you pasted does not. Write down every pasted copy before you click regenerate.

Then look outside Railway. Vercel, GitHub Actions, a laptop .env, a Supabase edge function secret, and a BI tool that uses DATABASE_PUBLIC_URL all keep the old password until you replace it. Regenerating on Railway does not revoke those copies. The issuer has to revoke, or the old password has to stop working inside the database.

Review logs at the issuer, not only on Railway. Stripe's dashboard, OpenAI's usage, Clerk's API keys page, and the database's connection log are where a used secret shows up. Railway asked for that review in the disclosure. A quiet log is not proof the key was never copied. It is a reason to rotate anyway and then watch.

Postgres on Railway, the way they documented it

Railway published a PostgreSQL rotation guide on 22 October 2026 as part of the same response. If the database uses their PostgreSQL template, the path they want is Regenerate in the Database view.

Before that click, their guide asks for three checks. Find every consumer on the variables tab. Services that reference the Postgres variables pick up the new password on their next deploy. Services with a pasted connection string do not. Check public networking. If public access is on, the credential works from the internet through the TCP proxy until you rotate, so turn public access off unless something outside the project needs it. List external consumers of DATABASE_PUBLIC_URL: local development, CI, and anything on another host. Those need the new value pasted by hand.

Regenerate itself is in the PostgreSQL service, Database tab, Config, Connection. Railway changes the password inside Postgres and updates POSTGRES_PASSWORD. In the template, PGPASSWORD, DATABASE_URL, and DATABASE_PUBLIC_URL reference that password, so they change too. The database restarts. Open sessions drop, which is the point if one of them was not yours. Your app cannot reconnect until you redeploy it. Regenerate redeploys the database, not the other services.

Their guide is explicit about the footgun. Do not edit POSTGRES_PASSWORD or PGPASSWORD by hand, and do not run ALTER USER yourself. Each changes only one side and leaves the variable and the database out of sync.

text

PostgreSQL service → Database → Config → Connection → Regenerate

After regenerate, redeploy every service that uses the database. Replace pasted connection strings with a reference so the next rotation reaches them. Update external consumers with the new DATABASE_PUBLIC_URL. A healthcheck that talks to the database fails a deploy that still has the old password. It will not notice a service you forgot, because that service was not redeployed.

If you do not suspect a leak and you need zero downtime, Railway points at a dual-role flow in a separate guide. If you suspect a leak, they say to regenerate first. Cutting off the old credential matters more than the short outage.

What this incident does not ask you to rotate

A publishable key is supposed to be in the browser. Rotating it because a host might have seen it does not close an open table.

  • Supabase's anon JWT and the newer sb_publishable_ key identify the project. Grants and row level security decide what they can read. The anon key post is that problem.
  • Clerk's pk_test_ and pk_live_ keys are the publishable key. CLERK_SECRET_KEY is the secret. The Clerk post separates them. A bare sk_live_ in a bundle is Stripe's prefix unless it sits on a Clerk assignment.
  • Firebase's web API key identifies the project. Security Rules decide reads. The key is not the secret.

If one of those publishable values was only ever public, Railway's disclosure is not a reason to treat it as a stolen master key. If the secret form of the same product was in Railway, rotate the secret.

A key compiled into client JavaScript is a separate leak. It does not require a host incident. NEXT_PUBLIC_ variables, and the env key in next.config, are bundled. The client bundle post is the search. The free scan looks for the same shapes in the live page, including an OpenAI key and a Clerk secret assignment. It will not tell you that Railway stored a variable. That question is answered in Railway's dashboard, then at the provider.

Order of operations

Do this in order so you do not lock yourself out and then discover the old key still works somewhere else.

  1. List every sensitive variable on every Railway service, and every place a copy might live.
  2. Regenerate Railway Postgres with their Database → Config → Regenerate control if you run their template. Redeploy consumers. Update DATABASE_PUBLIC_URL users.
  3. At each issuer, revoke the old API key and create a new one. Stripe secret and webhook signing secret, OpenAI, Clerk secret, Supabase service role, Neon password.
  4. Put the new values only in server-side env. Redeploy. Confirm the old value no longer authenticates.
  5. Read issuer logs for use of the old key. Then search the built client for the prefixes in the bundle post.

The checklist is the same five steps without this narrative. The free scan is step five against a URL, not a substitute for steps one through four.