Prepare, respond and recover
Secret rotation: verify consumers and revoke old credentials
How can a small team prove a secret rotation is complete?
A secret rotation is complete when every known consumer uses the replacement and the issuer has revoked the old credential.
Railway’s 22 October report says some variables in at most 1.85% of workspaces may have been compromised, but it could not identify the specific variables. Record which services you checked, the issuer’s confirmation that the old credential was revoked, and the log review without copying secret values.
A secret rotation is complete when every known consumer uses the replacement and the issuer has revoked the old credential. Railway’s preliminary report says some environment variables in at most 1.85% of workspaces may have been compromised, but it could not identify which specific variables were affected. 1Reference 1: Preliminary Report: Oct 21st Security Incident
The report recommends that customers who signed up before 21 October rotate credentials stored on Railway. At 19:18 UTC on 23 October, Railway said workspaces outside its higher-risk group were not confirmed safe. It also said customers who had already rotated since containment did not need to repeat the work for that update. At 22:06 UTC, Railway marked the platform incident resolved and advised customers who had not already rotated to replace sensitive credentials stored in Railway, revoke the old credentials and redeploy dependent services. 1Reference 1: Preliminary Report: Oct 21st Security Incident 2Reference 2: Railway Platform Outage — Resolved
For a small team, the work is proving that the old credential no longer works everywhere it was used. Changing a value in one dashboard does not show which services, build jobs or external tools have picked up the replacement.
Follow the credential from issuer to every service
Start with a short inventory. For each credential, record its name and issuer. Add the Railway project and environment, each service that uses it, any use outside Railway, and the person responsible for the change. Keep the value itself out of tickets, spreadsheets and evidence records.
This map matters because Railway variables can be available during builds, in running deployments, through the railway run command and in the Railway shell. Changes are staged and must be reviewed and deployed before they take effect. 3Reference 3: Using Variables
Make the old credential stop working
Use the issuer’s documented process to create a replacement. Depending on the credential, that may mean a new service key, a changed database password, a replacement private registry token or a reset integration credential.
Update each Railway variable and every copy used by an automated build or deployment job, a local operator tool, or a service outside the project. Review and deploy the pending changes. Then check that each dependent service starts, authenticates and passes a relevant health check with the replacement.
Finally, revoke the old credential at the system that issued it and record confirmation. Updating a dashboard value or restarting a service does not revoke a credential that remains valid at its issuer. OWASP lists rotation and revocation as separate lifecycle steps; potentially compromised credentials should be revoked. 4Reference 4: Secrets Management Cheat Sheet
Think of a building changing its locks after a key goes missing. The work is not complete until everyone who needs access has the new key and the lost one no longer opens a door.
Leave evidence without copying secrets
Railway advised users to review logs and later asked them to check provider activity for suspicious use. Review relevant Railway audit and deployment logs, plus activity from the systems that issued the credentials. Record the period reviewed, who checked it, any follow-up, and the deployment or health check that confirmed each service works. 1Reference 1: Preliminary Report: Oct 21st Security Incident 2Reference 2: Railway Platform Outage — Resolved
The record can show an owner for each credential, which consumers were updated, which checks passed, when the old credential was revoked, and what the log review covered. Leave out secret values so the evidence does not become another place where they can leak.