Writes done Right : Atomicity and Idempotency with Redis, Lua, and Go

Source: Dev.to

Introduction

Life would have been easy if the world were filled with monolithic functions—simple functions that execute once, and if they crash, we could try again.
But we build distributed systems, and life isn’t that easy (though it is fun).

A classic scenario: a user clicks “Pay Now.” The backend must:

1. Deduct the balance from the user’s wallet (Postgres).
2. Publish an event to send a confirmation email and notify the warehouse (Redis/Kafka).

If the database commits but the network fails before the event is published, the user is charged but the email is never sent, and the warehouse never ships the item. Reversing the process leads to the opposite problem. This is the Dual Write Problem, a silent killer of data integrity in micro‑services.

To solve it we need two architectural pillars:

• Atomicity – ensure database writes and event publications happen together.
• Idempotency – guarantee that repeated clicks result in a single charge.

We’ll build a robust backend using Go, Redis, and Postgres, implementing the Transactional Outbox Pattern for atomicity and using Redis for idempotency.

The complete, runnable source code is available on GitHub:
HERE

The Transactional Outbox Pattern

We must stop treating the database and the message broker as separate entities during a request. Since Redis cannot participate in a Postgres transaction, we bring the message queue to the database.

Transactional Outbox Pattern: instead of publishing directly to Redis, insert the message into a local SQL table (outbox) within the same transaction that modifies business data. A background worker later delivers these messages to Redis.

Architecture Overview

Database Schema

-- Business Table: stores the actual state
CREATE TABLE orders (
    id UUID PRIMARY KEY,
    user_id UUID NOT NULL,
    amount INT NOT NULL,
    status VARCHAR(50) DEFAULT 'PENDING',
    created_at TIMESTAMP DEFAULT NOW()
);

-- Outbox Table: stores the intent to publish
CREATE TABLE outbox (
    id UUID PRIMARY KEY,
    event_type VARCHAR(255) NOT NULL,   -- e.g., "order.created"
    payload JSONB NOT NULL,              -- data to publish
    status VARCHAR(50) DEFAULT 'PENDING',
    created_at TIMESTAMP DEFAULT NOW()
);

Go Implementation – CreateOrder

type Order struct {
    ID     uuid.UUID `json:"id"`
    UserID uuid.UUID `json:"user_id"`
    Amount int       `json:"amount"`
}

func CreateOrder(ctx context.Context, db *sql.DB, order Order) error {
    // 1. Start the transaction (atomicity boundary)
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        return fmt.Errorf("failed to begin transaction: %w", err)
    }
    defer tx.Rollback()

    // 2. Insert the business record
    _, err = tx.ExecContext(ctx,
        `INSERT INTO orders (id, user_id, amount) VALUES ($1, $2, $3)`,
        order.ID, order.UserID, order.Amount)
    if err != nil {
        return fmt.Errorf("failed to insert order: %w", err)
    }

    // 3. Insert the outbox record
    payload, err := json.Marshal(order)
    if err != nil {
        return fmt.Errorf("failed to marshal payload: %w", err)
    }

    _, err = tx.ExecContext(ctx,
        `INSERT INTO outbox (id, event_type, payload) VALUES ($1, $2, $3)`,
        uuid.New(), "order.created", payload)
    if err != nil {
        return fmt.Errorf("failed to insert outbox event: %w", err)
    }

    // 4. Commit the transaction
    if err := tx.Commit(); err != nil {
        return fmt.Errorf("failed to commit transaction: %w", err)
    }

    return nil
}

This function touches only Postgres. Both the order and the outbox entry are persisted together—all or nothing (Atomicity).

The Guard: Implementing Idempotency

In distributed systems, network failures can cause clients to retry requests, risking duplicate processing. We use Idempotency Keys (e.g., a UUID sent in the Idempotency-Key header) and store them in Redis, which offers fast atomic operations.

Three States of an Idempotency Key

Why Lua?

A naïve check‑then‑set approach can race:

// BAD CODE: DO NOT USE
if redis.Get(key) == "" {
    redis.Set(key, "PENDING")
    // ... process ...
}

Two concurrent requests could both see the key as empty and proceed, causing duplicate charges. Redis Lua scripts execute atomically, eliminating this race.

Lua Script (script.lua)

-- script.lua
local key = KEYS[1]
local pending_status = ARGV[1] -- "PENDING"
local ttl = ARGV[2]            -- expiration in seconds

-- 1. Check if key exists
local value = redis.call("GET", key)

-- 2. If it exists, return the value (could be "PENDING" or final JSON)
if value then
    return value
end

-- 3. If not, lock it with "PENDING" and a TTL
redis.call("SET", key, pending_status, "EX", ttl)
return nil

The TTL acts as a “dead‑man’s switch”: if the server crashes after setting PENDING, the key expires automatically, allowing the client to retry.

HTTP Handler with Idempotency Guard

func HandleCreateOrder(w http.ResponseWriter, r *http.Request, db *sql.DB, rdb *redis.Client) {
    // 1. Get Idempotency Key
    idempotencyKey := r.Header.Get("Idempotency-Key")
    if idempotencyKey == "" {
        http.Error(w, "Missing Idempotency-Key", http.StatusBadRequest)
        return
    }

    // 2. Execute Lua guard
    ctx := r.Context()
    val, err := rdb.Eval(ctx, luaScript, []string{idempotencyKey}, "PENDING", 60).Result()
    if err != nil && err != redis.Nil {
        http.Error(w, "Internal Server Error", http.StatusInternalServerError)
        return
    }

    // CASE A: Another request is processing
    if val == "PENDING" {
        http.Error(w, "Request is processing, please retry shortly", http.StatusConflict)
        return
    }

    // CASE B: Request already completed
    if val != nil {
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(fmt.Sprintf("%v", val)))
        return
    }

    // CASE C: New request – lock acquired
    orderID := uuid.New()
    order := Order{ID: orderID, UserID: uuid.New(), Amount: 1000}

    // Call the atomic transaction
    if err := CreateOrder(ctx, db, order); err != nil {
        // Remove lock on failure
        rdb.Del(ctx, idempotencyKey)
        http.Error(w, "Transaction failed", http.StatusInternalServerError)
        return
    }

    // 3. Store final result in Redis (cached for 24h)
    response := fmt.Sprintf(`{"status":"success","order_id":"%s"}`, orderID)
    rdb.Set(ctx, idempotencyKey, response, 24*time.Hour)

    w.Header().Set("Content-Type", "application/json")
    w.Write([]byte(response))
}

Now we have:

• Database protected by SQL transactions (Atomicity).
• API protected by Redis Lua scripts (Idempotency).

The Background Worker

The CreateOrder function leaves an event in the outbox table. A background worker (the “Courier”) polls this table and publishes messages to Redis.

Scaling the Worker

When multiple service replicas run, they could all fetch the same pending events. To avoid duplicate processing we use PostgreSQL’s FOR UPDATE SKIP LOCKED clause, which locks rows for a worker and skips those already locked by others.

Worker Implementation

func StartOutboxWorker(ctx context.Context, db *sql.DB, rdb *redis.Client) {
    ticker := time.NewTicker(500 * time.Millisecond)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            processBatch(ctx, db, rdb)
        }
    }
}

func processBatch(ctx context.Context, db *sql.DB, rdb *redis.Client) {
    // 1. Start a transaction
    tx, err := db.BeginTx(ctx, nil)
    if err != nil {
        log.Printf("Worker failed to begin tx: %v", err)
        return
    }
    defer tx.Rollback()

    // 2. Fetch pending events with SKIP LOCKED
    rows, err := tx.QueryContext(ctx, `
        SELECT id, event_type, payload
        FROM outbox
        WHERE status = 'PENDING'
        ORDER BY created_at ASC
        LIMIT 10
        FOR UPDATE SKIP LOCKED
    `)
    if err != nil {
        log.Printf("Worker query failed: %v", err)
        return
    }
    defer rows.Close()

    for rows.Next() {
        var id uuid.UUID
        var eventType string
        var payload []byte

        if err := rows.Scan(&id, &eventType, &payload); err != nil {
            continue
        }

        // 3. Publish to Redis
        if err := rdb.Publish(ctx, eventType, payload).Err(); err != nil {
            log.Printf("Failed to publish event %s: %v", id, err)
            return // transaction will roll back, event stays PENDING
        }

        // 4. Mark as processed
        _, err = tx.ExecContext(ctx,
            "UPDATE outbox SET status = 'PROCESSED' WHERE id = $1", id)
        if err != nil {
            return
        }
    }

    // 5. Commit the batch
    if err := tx.Commit(); err != nil {
        log.Printf("Worker failed to commit: %v", err)
    }
}

Robustness guarantees

• At‑Least‑Once Delivery – If publishing fails, the transaction rolls back, leaving the event PENDING for a later retry.
• Concurrency Safety – SKIP LOCKED lets many workers run in parallel without processing the same event twice.

Conclusion

By combining the Transactional Outbox Pattern with Idempotency (via Redis Lua scripts), we regain the simplicity of a monolithic function while operating in a distributed environment.

• Database writes and event publications are atomic.
• Repeated client requests are safely deduplicated.
• A background worker reliably delivers outbox events, scaling gracefully.

Reliability isn’t about hoping nothing breaks; it’s about building systems that handle breakages gracefully.