Building a Production-Ready Webhook Delivery System in 5 Minutes

Source: Dev.to

How It Works: The Flow

Key Features

• ⚡ Instant response (202 Accepted)
• 🔄 Parallel delivery to all subscribers
• 🔐 HMAC SHA‑256 signatures on every webhook
• ♻️ Auto‑retry with exponential backoff (1 s, 2 s, 4 s)
• 📊 Health monitoring and auto‑disable after 10 failures

What Are Outgoing Webhooks?

Most developers are familiar with receiving webhooks from services like Stripe, GitHub, or Slack. Outgoing webhooks (also called “reverse webhooks”) let your application notify external services when events occur. When something happens in your system—a user signs up, an order ships, a payment completes—your app sends HTTP POST requests to registered webhook URLs.

This is crucial for building integrations, enabling real‑time notifications, and allowing customers to extend your platform.

The Challenge: Building It Right Is Hard

Building a robust webhook delivery system from scratch involves solving many complex problems:

• URL Verification – validating that webhook URLs are legitimate
• Security – HMAC signatures to prevent tampering
• Reliability – handling down endpoints gracefully
• Scale – delivering to thousands of subscribers without blocking
• Monitoring – delivery stats, failure tracking, auto‑disabling broken webhooks
• Retry Logic – exponential backoff, max attempts, timeout handling

Queue‑Based Architecture: The Secret Sauce

The key to scalable webhook delivery is decoupling event triggering from delivery.

// 1. Store the event for audit trail
await conn.insertOne('events', eventData);

// 2. Mark all matching webhooks with the event ID
await conn.updateMany(
  'webhooks',
  { status: 'active', $or: [{ events: eventType }, { events: '*' }] },
  { $set: { pendingEventId: eventData.id } }
);

// 3. Queue ALL webhooks in one atomic operation
await conn.enqueueFromQuery(
  'webhooks',
  { status: 'active', pendingEventId: eventData.id },
  'webhook-delivery'
);

The magic is in enqueueFromQuery. Instead of loading webhooks into memory and looping through them, it:

• Queues everything in a single database operation
• Handles thousands of webhooks without memory overhead
• Returns immediately (202 Accepted)
• Processes deliveries in parallel via worker functions

This architecture delivers instant response times even with massive subscriber lists.

Security First: HMAC Signing

Every webhook payload includes a cryptographic signature so receivers can verify authenticity.

function generateSignature(payload, secret) {
  const timestamp = Math.floor(Date.now() / 1000);
  const sigBasestring = `${timestamp}.${payload}`;
  const signature = crypto
    .createHmac('sha256', secret)
    .update(sigBasestring, 'utf8')
    .digest('hex');
  return { signature: `v1=${signature}`, timestamp };
}

Headers sent with each webhook

• X-Webhook-Signature: HMAC SHA‑256 signature
• X-Webhook-Timestamp: Unix timestamp (prevents replay attacks)
• X-Webhook-Id: Subscription ID

Receivers validate signatures and reject old timestamps (> 5 minutes) to prevent replay attacks.

Automatic URL Verification

Before accepting webhook registrations, the system verifies URLs using industry‑standard methods:

• Stripe‑style verification – sends a test payload with a verification token and expects HTTP 200.
• Slack‑style challenge – sends a random challenge string and expects it echoed back in the response.

Verification runs asynchronously in a worker queue, so registration returns immediately while verification proceeds in the background.

Smart Retry Logic

Failed deliveries trigger automatic retries with exponential backoff.

{
  "maxRetries": 3,
  "backoffIntervals": ["1s", "2s", "4s"],
  "timeout": "10s",
  "autoDisableAfter": 10
}

A cron job runs every 30 minutes to retry webhooks that have been failing for over an hour, preventing hammering of broken endpoints.

Event Flexibility

Unlike some systems that require predefined event types, this system supports any event name.

# E‑commerce events
POST /events/trigger/order.placed

# IoT events
POST /events/trigger/sensor.temperature.high

# Custom business events
POST /events/trigger/report.generated

Subscribers can register for specific events or use "*" as a wildcard to receive everything.

Production‑Grade Monitoring

The system tracks detailed statistics for each webhook.

{
  "deliveryCount": 1247,
  "consecutiveFailures": 0,
  "lastDeliveryAt": "2025-01-15T10:30:00Z",
  "lastDeliveryStatus": "success",
  "status": "active"
}

After 10 consecutive failures, webhooks are automatically disabled to conserve resources. Users can manually retry with POST /webhooks/:id/retry.

The Complete API

A full CRUD API is provided for webhook management.

# Create webhook (with automatic verification)
POST /webhooks
{
  "clientId": "customer-123",
  "url": "https://example.com/webhook",
  "events": ["order.placed", "order.shipped"],
  "verificationType": "stripe"
}

# List webhooks
GET /webhooks?status=active&event=order.placed

# Trigger events (queues delivery to all subscribers)
POST /events/trigger/order.placed
{
  "orderId": "123",
  "total": 99.99,
  "customer": "john@example.com"
}

# Check delivery stats
GET /webhooks/:id/stats

# Manually retry failed webhook
POST /webhooks/:id/retry

Idempotent Registration

The system uses clientId + url as a composite key for upsert logic:

• First registration creates the webhook.
• Subsequent registrations with the same clientId + url update it.
• Prevents duplicate webhooks.
• Preserves HMAC secrets across updates.
• Ideal for multi‑tenant SaaS applications.

Real‑World Integration

The system integrates seamlessly with popular workflow tools:

• n8n – create a webhook trigger, register the URL, and workflows activate automatically.
• Zapier – use “Webhooks by Zapier” with Catch Hook, register the URL with verificationType: "stripe", and your Zaps come alive.
• Custom apps – implement signature verification in any language and start receiving events.

See It In Action

Want to try it yourself? The complete implementation is available as a Codehooks.io template.