Securely Triggering a Private Jenkins Using GitHub Webhooks, API Gateway (Public), and Lambda (Private)

Source: Dev.to

The Idea Behind the Architecture

The core idea is simple:

1. API Gateway is the only public‑facing component.
2. Lambda and Jenkins stay private inside a VPC.
3. Everything is validated before Jenkins is triggered.

GitHub (Public)
   |
   | HTTPS Webhook
   v
API Gateway (Public REST API)
   |
   v
AWS Lambda (Private – inside VPC)
   |
   v
Jenkins (Private – EC2 / VPC)

Result

• Jenkins never sees Internet traffic.
• Lambda becomes the security gate.
• Only verified GitHub events trigger builds.

What We’re Using

• API Gateway (REST API) – public webhook endpoint.
• AWS Lambda (inside a VPC) – validates the payload and forwards the request.
• Private Jenkins – running inside the same VPC.
• GitHub HMAC signature verification – ensures payload integrity.
• No fancy services – just the right components in the right places.

Step‑by‑Step Guide

1️⃣ Set Up Jenkins (Private)

• Run Jenkins on an EC2 instance (or VM) inside a VPC.
• Place it in a private subnet with no public IP.
• Security Group
  • Inbound: allow traffic only from the Lambda security group.
  • Outbound: allow HTTPS if Jenkins needs external access.

At this point Jenkins is completely isolated from the Internet.

2️⃣ Create the Lambda Function (Private)

• Attach the function to private subnets.
• Use a security group that allows outbound HTTPS (to reach Jenkins).

3️⃣ Create API Gateway (Public Entry Point)

⚠️ Use a REST API, not an HTTP API.
REST API gives full control over request validation and headers.

1. API Gateway → Create API → Choose REST API.
2. Create a resource (e.g., /).
3. Add a POST method.
4. This API will be the only thing exposed to the Internet.

4️⃣ Connect API Gateway to Lambda

• In the POST method’s Integration Request:
  • Integration type: Lambda Function
  • Select the Lambda you created.
• Deploy the API (e.g., to stage dev).

Enable Lambda Proxy Integration – it forwards headers, body, and the raw request payload unchanged.

5️⃣ Method Request Configuration (The Gotcha)

1. Open API Gateway → Resources → POST → Method Request.
2. Set:

Why this matters
Without the request validator, API Gateway may drop headers such as X‑Hub‑Signature‑256. The signature verification in Lambda would then always fail.

3. Redeploy the API after changing the setting.

6️⃣ Configure the GitHub Webhook

In your GitHub repository:

Settings → Webhooks → Add webhook

GitHub will now send:

• The raw JSON payload.
• The X‑Hub‑Signature‑256 header.

What the Lambda Actually Does

The Lambda function acts as a security gate between GitHub and Jenkins. It:

1. Reads the raw webhook payload.
2. Verifies the GitHub HMAC signature.
3. Checks the branch (or other criteria).
4. Triggers a Jenkins job privately.

Lambda Code (Production‑Ready)

import json
import hmac
import hashlib
import base64
import urllib.parse
import requests
from requests.auth import HTTPBasicAuth

# ----------------------------------------------------------------------
# Configuration
# ----------------------------------------------------------------------
GITHUB_SECRET = b"demo-github-secret"

JENKINS_URL   = "https://jenkins.example.com"
JENKINS_USER  = "demo-user"
JENKINS_TOKEN = "demo-api-token"

JOB_URL = f"{JENKINS_URL}/job/demo-pipeline/buildWithParameters"
# ----------------------------------------------------------------------

def extract_payload(event):
    """
    Preserve the exact raw body GitHub signed.
    """
    # Normalise header keys to lower‑case
    headers = {k.lower(): v for k, v in (event.get("headers") or {}).items()}
    body = event.get("body", "")

    # GitHub may send the payload base64‑encoded (API Gateway default)
    if event.get("isBase64Encoded"):
        raw_body = base64.b64decode(body)
    else:
        raw_body = body.encode("utf-8")

    # Some GitHub integrations wrap the JSON in a `payload=` form‑urlencoded string
    if raw_body.startswith(b"payload="):
        raw_body = urllib.parse.unquote_to_bytes(
            raw_body.decode().replace("payload=", "", 1)
        )
    return raw_body, headers

def verify_signature(raw_body, headers):
    """
    Verify the HMAC SHA‑256 signature sent by GitHub.
    """
    signature_header = headers.get("x-hub-signature-256")
    if not signature_header:
        raise ValueError("Missing X-Hub-Signature-256 header")

    # Header format: sha256=abcdef...
    try:
        _, received_sig = signature_header.split("=")
    except ValueError:
        raise ValueError("Malformed signature header")

    mac = hmac.new(GITHUB_SECRET, raw_body, hashlib.sha256)
    expected_sig = mac.hexdigest()

    if not hmac.compare_digest(expected_sig, received_sig):
        raise ValueError("Signature verification failed")
    return True

def trigger_jenkins(payload):
    """
    Call Jenkins with the appropriate parameters.
    """
    data = {
        "token": JENKINS_TOKEN,
        # Example: forward the branch name
        "BRANCH": payload.get("ref", "").split("/")[-1],
    }
    response = requests.post(
        JOB_URL,
        data=data,
        auth=HTTPBasicAuth(JENKINS_USER, JENKINS_TOKEN),
        verify=False,  # set to True with proper certs
        timeout=10,
    )
    response.raise_for_status()
    return response.text

def lambda_handler(event, context):
    try:
        raw_body, headers = extract_payload(event)
        verify_signature(raw_body, headers)

        # Parse JSON *after* verification
        payload = json.loads(raw_body)

        # Optional: filter branches, actions, etc.
        if payload.get("ref") != "refs/heads/main":
            return {"statusCode": 200, "body": "Ignored non‑main branch"}

        trigger_jenkins(payload)

        return {"statusCode": 200, "body": "Jenkins job triggered"}
    except Exception as e:
        # Log the error (CloudWatch) and return a 400/500 as appropriate
        print(f"Error: {e}")
        return {"statusCode": 400, "body": str(e)}

Key points in the code

• extract_payload preserves the exact byte sequence GitHub signed (including handling of payload= form‑urlencoded bodies).
• verify_signature uses hmac.compare_digest to avoid timing attacks.
• trigger_jenkins calls the Jenkins job over the private VPC network using basic auth (or an API token).

Recap

By keeping Jenkins completely off the Internet and letting API Gateway + Lambda act as the gatekeeper, you get a secure, production‑ready solution for GitHub‑triggered builds. The only “gotcha” is enabling the request validator in API Gateway so the signature header isn’t stripped away. Once that’s set, the flow works flawlessly.

Overview

This document explains a production‑ready AWS Lambda function that acts as a secure bridge between GitHub webhooks and a private Jenkins server.

It:

1. Extracts the raw webhook payload.
2. Verifies the GitHub signature.
3. Filters events (only the main branch).
4. Triggers the Jenkins pipeline over the internal network.

Verifying the GitHub Signature

def verify_signature(raw_body, headers):
    signature = headers.get("x-hub-signature-256")
    if not signature:
        return False

    expected = "sha256=" + hmac.new(
        GITHUB_SECRET,
        raw_body,
        hashlib.sha256
    ).hexdigest()

    # Securely compare the two values to prevent timing attacks
    return hmac.compare_digest(signature, expected)

• If the x-hub-signature-256 header is missing, the request is rejected immediately.
• The expected signature is recomputed from the raw bytes of the payload.
• hmac.compare_digest provides a constant‑time comparison.

Main Lambda Handler

def lambda_handler(event, context):
    # 1️⃣ Extract and verify first – never parse JSON before validation
    raw_body, headers = extract_payload(event)

    if not verify_signature(raw_body, headers):
        return {"statusCode": 401, "body": "Invalid GitHub signature"}

    # 2️⃣ Safe to parse JSON now
    payload = json.loads(raw_body.decode("utf-8"))

    # 3️⃣ Process only pushes to the main branch
    ref = payload.get("ref", "")
    if not ref.endswith("/main"):
        return {"statusCode": 200, "body": "Not main branch"}

    # 4️⃣ Trigger Jenkins (privately)
    session = requests.Session()
    session.auth = HTTPBasicAuth(JENKINS_USER, JENKINS_TOKEN)

    # Handle Jenkins CSRF protection
    crumb = session.get(f"{JENKINS_URL}/crumbIssuer/api/json").json()
    crumb_header = {crumb["crumbRequestField"]: crumb["crumb"]}

    response = session.post(
        JOB_URL,
        headers=crumb_header,
        params={"BRANCH": "main"},
        timeout=10
    )

    return {
        "statusCode": response.status_code,
        "body": "Jenkins pipeline triggered"
    }

• Invalid requests are blocked before any further processing, so Jenkins never sees them.
• The JSON payload is parsed only after the signature check passes.
• The function returns a 200 response for non‑main branches, keeping the webhook happy while doing nothing.

Understanding the Lambda Code

Full Production‑Ready Lambda Code

import json
import hmac
import hashlib
import base64
import urllib.parse
import requests
from requests.auth import HTTPBasicAuth

# ── Configuration ────────────────────────────────────────────────────────
GITHUB_SECRET = b"demo-github-secret"

JENKINS_URL   = "https://jenkins.example.com"
JENKINS_USER  = "demo-user"
JENKINS_TOKEN = "demo-api-token"

JOB_URL = f"{JENKINS_URL}/job/demo-pipeline/buildWithParameters"
# ────────────────────────────────────────────────────────────────────────

def extract_payload(event):
    """
    Returns:
        raw_body (bytes): the exact bytes that GitHub sent.
        headers (dict): lower‑cased header dict for easy lookup.
    """
    headers = {k.lower(): v for k, v in (event.get("headers") or {}).items()}
    body = event.get("body", "")

    # API Gateway may Base64‑encode the body
    if event.get("isBase64Encoded"):
        raw_body = base64.b64decode(body)
    else:
        raw_body = body.encode("utf-8")

    # GitHub can send payload as `payload=...` (url‑encoded)
    if raw_body.startswith(b"payload="):
        raw_body = urllib.parse.unquote_to_bytes(
            raw_body.decode().replace("payload=", "", 1)
        )

    return raw_body, headers

def verify_signature(raw_body, headers):
    """
    Re‑creates the HMAC‑SHA256 signature and compares it with the header.
    Returns True if the signature matches, False otherwise.
    """
    signature = headers.get("x-hub-signature-256")
    if not signature:
        return False

    expected = "sha256=" + hmac.new(
        GITHUB_SECRET,
        raw_body,
        hashlib.sha256
    ).hexdigest()

    return hmac.compare_digest(signature, expected)

def lambda_handler(event, context):
    raw_body, headers = extract_payload(event)

    if not verify_signature(raw_body, headers):
        return {"statusCode": 401, "body": "Invalid GitHub signature"}

    payload = json.loads(raw_body.decode("utf-8"))

    # Only act on pushes to the main branch
    ref = payload.get("ref", "")
    if not ref.endswith("/main"):
        return {"statusCode": 200, "body": "Not main branch"}

    # Authenticate to Jenkins
    session = requests.Session()
    session.auth = HTTPBasicAuth(JENKINS_USER, JENKINS_TOKEN)

    # Obtain CSRF crumb
    crumb = session.get(f"{JENKINS_URL}/crumbIssuer/api/json").json()
    crumb_header = {crumb["crumbRequestField"]: crumb["crumb"]}

    # Trigger the pipeline
    response = session.post(
        JOB_URL,
        headers=crumb_header,
        params={"BRANCH": "main"},
        timeout=10
    )

    return {
        "statusCode": response.status_code,
        "body": "Jenkins pipeline triggered"
    }