How Blacksmith used Tailscale Services to work around ISP degradation

Source: Tailscale Blog

TL;DR

We built a transparent proxy that routes GitHub traffic through an alternate network path with direct GitHub peering, giving our runners defence‑in‑depth against ISP routing failures. No code changes required for customers—GitHub Actions interact with GitHub exactly as they used to. This proxy, called Blacksmith, runs as a Tailscale service and automatically fails over to the public internet if the direct peering path is unavailable.

Background

• Problem: ISP routing failures can cause intermittent connectivity issues for CI/CD pipelines that rely on GitHub.
• Goal: Provide a reliable, transparent network path that does not require changes to existing GitHub Actions workflows.

Design Overview

1. Transparent Proxy

   • Deployed as a Tailscale service.
   • Intercepts outbound GitHub traffic from runners.

2. Direct Peering

   • Establishes a dedicated peering connection with GitHub’s network.
   • Bypasses typical ISP routing.

3. Failover Mechanism

   • Monitors the health of the peering link.
   • Automatically falls back to the public internet if the peering link degrades.

Implementation Details

• Tailscale Service: Runs on a lightweight VM within the same VPC as the runners.
• IP Routing: Uses iptables rules to redirect traffic destined for github.com to the proxy.
• Health Checks: Periodic TCP probes to GitHub’s endpoints determine link health.
• Configuration: No changes needed in the runner’s .github/workflows files; the proxy is invisible to the CI jobs.

Benefits

• Zero Code Changes: Existing pipelines continue to work unchanged.
• Improved Reliability: Reduces the impact of ISP‑level routing issues.
• Observability: Tailscale’s admin console provides metrics on proxy health and traffic.

Getting Started

1. Deploy Blacksmith: Follow the Terraform module provided in the repository.
2. Enable Tailscale on Runners: Ensure runners are part of the same Tailscale network.
3. Verify Connectivity: Use the provided health‑check script to confirm the peering path is active.

Future Work

• Expand support to other major code hosting providers.
• Add optional TLS termination for additional security layers.
• Provide a UI dashboard for real‑time traffic analytics.