Mitigating the Two Generals Problem in Django with Idempotence

Source: Dev.to

A Fintech Architecture Perspective

When building payment systems, you eventually face a terrifying question:

Did we charge the customer or not?

The worst part?
Sometimes, the system genuinely doesn’t know.

This is not a bug. It’s a distributed‑systems reality described by the Two Generals Problem: two parties cannot guarantee agreement over an unreliable network.

In fintech this becomes:

Merchant → Backend → Payment Provider

• Charge succeeds
• Network drops
• Backend never receives confirmation
• Client retries

Now what?

• If you retry blindly, you double‑charge.
• If you don’t retry, you might lose revenue.

This is where idempotency becomes a core architectural pattern.

Why This Matters in Fintech

In international payment systems:

• Mobile networks are unstable
• Clients double‑click
• Workers crash mid‑transaction

If your backend is not idempotent you will:

• Double‑charge customers
• Create accounting inconsistencies
• Trigger reconciliation nightmares
• Lose merchant trust

Large providers like Stripe formalized idempotency because this problem is structural, not incidental.

The Goal: Deterministic Retries

We cannot guarantee exactly‑once execution over HTTP.
What we can guarantee:

The same request key always produces the same result.

That’s the architectural shift. Instead of solving uncertainty, we make retries safe.

Designing Idempotency in Django

This is not about a simple if exists check. It’s about enforcing correctness at the database boundary.

1. Require an Idempotency Key

Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000

The key represents one logical business operation.

2. Persist the Key with a Unique Constraint

# models.py
from django.db import models

class IdempotencyRecord(models.Model):
    key = models.CharField(max_length=255, unique=True)
    request_hash = models.CharField(max_length=64)
    response_body = models.JSONField(null=True, blank=True)
    response_status = models.IntegerField(null=True, blank=True)
    created_at = models.DateTimeField(auto_now_add=True)

Why this matters

• unique=True protects against concurrency.
• The database becomes the source of truth.
• Race conditions move from application logic to atomic constraints.

3. Hash the Request Payload

import hashlib

def hash_request(request):
    """Return a SHA‑256 hash of the raw request body."""
    return hashlib.sha256(request.body).hexdigest()

Why?
If someone reuses the same key with a different payload, that’s dangerous. We must detect:

• Same key
• Different business intent

and reject it. The hash is generated on the server when Django receives the request.

request payload → server‑side hash computation
                → database storage
                → hash comparison on retries
                → deterministic decision (replay or reject)

4. Atomic Handling

from django.db import transaction, IntegrityError
from django.http import JsonResponse
from .models import IdempotencyRecord

@transaction.atomic
def process_payment(request):
    key = request.headers.get("Idempotency-Key")
    if not key:
        return JsonResponse({"error": "Missing Idempotency-Key"}, status=400)

    request_hash = hash_request(request)

    try:
        # First time we see this key
        record = IdempotencyRecord.objects.create(
            key=key,
            request_hash=request_hash
        )
        first_request = True
    except IntegrityError:
        # Key already exists – fetch the existing row for update
        record = IdempotencyRecord.objects.select_for_update().get(key=key)
        first_request = False

    if not first_request:
        # Duplicate request – ensure payload matches
        if record.request_hash != request_hash:
            return JsonResponse({"error": "Payload mismatch"}, status=409)
        return JsonResponse(record.response_body, status=record.response_status)

    # ----- Side effect: external charge -----
    charge = external_payment_call()   # <-- implement your provider call

    response = {"payment_id": charge.id}

    # Store the successful response for future replays
    record.response_body = response
    record.response_status = 200
    record.save()

    return JsonResponse(response, status=200)

What this guarantees

• The first request executes the side effect (charge).
• Duplicate requests return the stored result.
• No duplicate charges.
• Safe client retries.

What This Achieves Architecturally

• Simulates exactly‑once semantics.
• Accepts eventual consistency.
• Moves correctness into a deterministic replay.
• Makes the system resilient to network failures.

It does not eliminate distributed uncertainty; it engineers around it.

Critical Fintech Edge Cases

1. DB Crash After External Charge

If the charge succeeds but the DB commit fails, you still have inconsistency.

Mitigation

• Use provider webhooks to confirm settlement.
• Run reconciliation jobs that compare external logs with internal records.
• Adopt a ledger‑based accounting model where every event is immutable.

2. Key Expiration

Idempotency tables grow indefinitely.

Solution

• Add a TTL/cleanup job.
• Typical retention:
  • 24‑48 h for payment operations.
  • Longer (weeks) for financial transfers that may be audited.

3. Observability

In production fintech systems:

• Every idempotency key is traceable (log the key with request ID).
• Every retry is logged with timestamps and outcome.
• Metrics track replay frequency, collision rate, and latency impact.

Idempotency without observability is incomplete.

Closing Thought

In fintech, correctness is non‑negotiable. Idempotent API design gives you a deterministic, auditable way to handle the inevitable uncertainty of distributed systems—turning a scary “Did we charge?” question into a predictable, safe‑to‑retry workflow.

Overview

Ess is not about writing clean code. It is about designing systems that behave deterministically under failure.

Failure scenarios

• Networks will drop.
• Clients will retry.
• Workers will crash.

Key principle

Idempotency is how you make your Django system financially safe in an unreliable world.