How I Normalized 74,000+ Scattered Coupon Feeds into a Single Real-Time API (And Built a Live Sandbox)
Source: Dev.to
Hey fellow devs, If you’ve ever dealt with affiliate networks like CJ, Awin, or Impact, you already know the pain. Each network speaks a completely different language. One gives you data in a bloated XML format, another uses a weirdly nested JSON structure, and the third one updates its endpoints whenever it feels like it. A few months ago, I got sick of writing separate custom parsers for every single network just to keep a coupon directory updated. So, I decided to build a unified middleware that does the heavy lifting: aggregating, parsing, de-duplicating, and normalizing tens of thousands of affiliate data points into a single, blazing-fast API. Today, I finally rolled out the live playground for it, and I wanted to share the architecture under the hood and how it works. The Problem: The “Affiliate Spaghetti Data” brand_id vs merchant_name: Network A calls it “Merchant XY”, Network B calls it “Merchant XY LLC”. Expiry Dates: Some networks use Unix timestamps, some use YYYY-MM-DD, and others just leave it null for ongoing deals. Tracking Links: Appending custom click parameters (subID or sid) without breaking the redirect chain is a nightmare. If your database can’t handle this normalization dynamically, your frontend will either render broken links or stale data. The Architecture: How It Works Here is the high-level flow of the engine: The Ingestion Layer: A fleet of cron jobs (Node.js/TypeScript) fetches raw data from master affiliate APIs at staggered intervals to prevent rate-limiting. The Normalization Pipeline: Every raw coupon object passes through a strict validation schema. Names are slugified, dates are unified to a standard ISO format, and expired tokens are instantly flagged. The Storage Strategy: Instead of blasting heavy relational JOIN queries every time a user types a word, everything is indexed efficiently so it handles full-text search instantly. The Frontend Challenge: Making It Look Like an API Dashboard I spent the last few days cooking the frontend. It uses Server-Side Rendering (SSR) for the initial payload so search engine bots don’t just see a blank loading spinner, but the search itself is fully reactive. When you type a keyword or a brand name (e.g., “travelup” or “hosting”), it instantly filters through the dataset with a 300ms debounce, updating the browser URL on the fly without refreshing the page. Here is a quick look at what the dashboard looks like right now: TypeScript const filterCoupons = (query: string, dataset: Coupon[]) => { const cleanQuery = query.toLowerCase().trim(); if (!cleanQuery) return dataset.slice(0, 20); // Default fallback
return dataset.filter(item => item.brand_name.toLowerCase().includes(cleanQuery) || item.coupon_description.toLowerCase().includes(cleanQuery) ); };
Check Out the Live Sandbox You can break it, search for random brands, test the response speeds, or download sample raw CSV/XML exports directly from the UI:https://feedico.io/live-coupon-feed Feedico Live Coupon Feed & Sandbox If you want to see a specific query example in action (like how it renders pre-filtered data via SSR), check out this live endpoint instance: What’s Next? I’d love to get some brutal feedback from the Dev.to community: How’s the initial page load speed for you? Would you prefer Webhooks over standard REST polling for real-time coupon updates? Let me know in the comments below! Appreciate you guys reading.