How We Eliminated Duplicate Orders at Scale: A Python Post-Mortem from EyecareWell

Source: Dev.to

Introduction — A Production Incident

EyecareWell operates a Shopify‑backed ecommerce platform alongside a Windows blue‑light filter application. As daily order volume increased, a production incident surfaced: duplicate webhook executions caused order duplication and inventory drift.

Incident Summary

• Duplicate order creation
• Inventory mismatch across systems
• Repeated webhook retries from Shopify

Root Cause Analysis

The webhook endpoint was synchronous and non‑idempotent. Shopify retries events when responses exceed timeout thresholds, leading to race conditions and duplicate writes.

Architecture Before

• Flask‑based webhook endpoint
• Direct DB writes
• No background processing
• No deduplication

Architecture After

• FastAPI async endpoint
• Redis‑backed idempotency
• Celery background workers
• Immediate HTTP acknowledgment

Key Python Implementation

if redis.exists(event_id):
    return {"status": "duplicate"}
redis.set(event_id, "processing", ex=3600)

Observability Improvements

• Structured logging
• Retry metrics
• Error alerting

Results

• 0 duplicate orders post‑fix
• 70 % reduction in webhook latency
• Stable inventory reconciliation

Lessons Learned

• Always assume webhooks will retry
• Idempotency is non‑optional
• Async processing improves reliability

Why This Matters

This architecture allows EyecareWell to scale safely while maintaining trust for health‑focused products.

Conclusion

The case study demonstrates how disciplined Python backend design solves real production issues in ecommerce systems.