Idempotency

Source: Dev.to

What is Idempotency?

An operation is idempotent if it can be applied multiple times without changing the result beyond the initial application. In the context of an API, this means that making the same call multiple times must not have any side effects on the business layer.

Why Systems Fail Without Idempotency

Idempotency Keys

The standard pattern for enforcing exactly‑once processing of a transaction is the use of Idempotency Keys.

Implementation Workflow

1. Key Generation – The client generates a unique identifier (e.g., UUID) for the operation.
2. Request Header – The key is sent in a custom header (e.g., X-Idempotency-Key).
3. Server‑Side Validation – If the key exists in the “processed” cache, the server returns the cached response immediately.
4. Processing New Keys – If the key is new, the server acquires a lock, processes the request, and stores the result before committing.

Client Side

// Generate UUID on first attempt
const idempotencyKey = uuidv4(); // "abc-123-def-456"

// Send request with key in header
fetch('/api/orders', {
  method: 'POST',
  headers: {
    'Idempotency-Key': idempotencyKey,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({ productId: 42, quantity: 1 })
});

// On retry (timeout/error), use THE SAME key
// The key doesn't change until operation succeeds or max retry is reached

Server Side in Go

func HandleCreateOrder(w http.ResponseWriter, r *http.Request) {
    idempotencyKey := r.Header.Get("Idempotency-Key")

    // Reject requests without key
    if idempotencyKey == "" {
        http.Error(w, "Idempotency-Key required", http.StatusBadRequest)
        return
    }

    // Check if already processed
    cachedResponse, found := checkKeyAlreadyProcessed(idempotencyKey)
    if found {
        // Return cached response
        w.WriteHeader(http.StatusCreated)
        w.Write(cachedResponse)
        return
    }

    // Process the order
    order, err := createOrder(r.Body)
    if err != nil {
        http.Error(w, "Failed to create order", http.StatusInternalServerError)
        return
    }

    // Store key for future retries
    storeProcessedKey(idempotencyKey, order, 24*time.Hour)

    w.WriteHeader(http.StatusCreated)
    json.NewEncoder(w).Encode(order)
}

By implementing idempotency, multiple requests with the same key won’t have any side effects.

Strategic Implementation Best Practices

Persistence Layer Choice

• Redis – Ideal for performance. Set a Time‑to‑Live (TTL) of 24–48 hours to prevent memory growth.
• Relational DB – Ideal for strict consistency. Store the key within the same transaction as the business logic to ensure atomicity.

Deterministic Responses

An idempotent retry should return the original status code. If the first request returned 201 Created, subsequent retries with the same key should also return 201 Created, not 200 OK or 409 Conflict. This keeps client‑side logic simple and consistent.

Scope of the Key

Keys should be scoped to the user or account. This prevents collisions where two different users might accidentally generate the same UUID (unlikely but possible in multi‑tenant environments).

The Read‑Modify‑Write Problem

A common mistake is failing to account for concurrency. In high‑traffic systems, two identical requests might reach your API at the exact same millisecond.

// ❌ CRITICAL BUG: Race Condition
const record = await db.idempotency.find(key);
if (!record) {
    // Both Request A and Request B can reach this line simultaneously
    await service.processTransaction(); 
}

Fix this with database‑level atomicity:

• SQL – Use a UNIQUE constraint on the idempotency‑key column and handle violation errors.
• Redis – Use the SET NX command to ensure only one worker can process the key.
• NoSQL – Use conditional updates or atomic “set‑if‑not‑exists” operations.

Conclusion

Idempotency is a main pillar of building resilient, fault‑tolerant distributed systems. By moving the responsibility of deduplication from business logic to a structured architectural pattern, we eliminate entire classes of bugs related to double‑spending and data corruption.