Railway Blocked by Google Cloud

Source: Hacker News

Railway Service Disruption (Major Outage)

Affected components: Edge Network

Updates

• Investigating – May 19, 22:29 UTC
  We are investigating a widespread service disruption affecting Railway. Users may be experiencing errors including “no healthy upstream”, “unconditional drop overload”, login failures, and inability to access the dashboard. We are actively investigating and will provide updates as soon as possible.

• Identified – May 19, 22:43 UTC
  We have identified the cause of the disruption. Access to our upstream cloud provider has been restored and we are working on a fix.

  The Railway dashboard and services running on our cloud infrastructure may continue to be affected until the fix is deployed.

• Identified – May 19, 23:37 UTC
  Google Cloud has blocked our account, making some Railway services unavailable. We have escalated this directly with Google.

  The Railway Platform team has since confirmed access to Google Cloud and is working on restoring access to all workloads. We have access to some of our Google Cloud–hosted infrastructure and are working to restore the rest of the service. We apologize for the disruption.

• Identified – May 20, 00:37 UTC
  We are working to restore the Google Cloud infrastructure that powers our dashboard, API, and internal network’s control plane. We are in direct contact with Google Cloud’s support team. We do not have an ETA at this time.

• Identified – May 20, 01:23 UTC
  We are continuing to work on restoring access to our workloads on Google Cloud. In parallel, our infrastructure team is evaluating alternative paths to bring affected services back online. We do not have an ETA at this time and will continue posting updates as we have them.