Why using Elasticsearch was bad because I needed real-time data retrieval, not just fast searching
Source: Dev.to
🚧 The problem I was actually solving
In production, our pain wasn’t search speed.
It was real‑time correctness.
We had workflows where:
- A user action needed to be reflected immediately
- State changes had to be visible within milliseconds
- “Eventually consistent” was not good enough to explain to customers or support teams
But at the time, I framed the problem incorrectly:
“Queries are slow → add Elasticsearch”
That framing cost us months.
🔍 What went wrong at scale
Elasticsearch did exactly what it promises:
- Fast, distributed search over indexed data.
What it does not promise:
- Strong real‑time guarantees
- Immediate consistency after writes
- Acting as a primary data source for transactional flows
In real production traffic we started seeing:
- Recently updated records missing from results
- Edge cases where writes succeeded but reads lagged
- Complex retry and refresh logic creeping into application code
Every bug felt “rare”, but together they were constant.
Non‑technical stakeholders didn’t care why it happened; they just saw data that felt unreliable.
⚖️ The trade‑off I underestimated
Elasticsearch trades correctness now for performance at scale.
We needed predictable reads after writes.
🧠 The lesson that stuck with me
The mistake wasn’t using Elasticsearch.
The mistake was using it to compensate for a poorly defined problem.
- Fast search ≠ real‑time data retrieval
- Indexing ≠ state management
- Scalability ≠ correctness
Once we re‑centered the architecture around:
- A primary, strongly consistent data store
- Clear read/write paths
- Search used only where search made sense
the system became calmer, incidents dropped, and engineers slept better.
🌱 How this changed how I design systems
Today, I’m much slower to introduce “powerful” infrastructure.
Before adding anything new, I ask:
- What guarantee does this system give me?
- What guarantee does it explicitly not give?
- What failure will I be debugging six months from now?
Most production pain doesn’t come from lack of tools; it comes from misaligned guarantees.
Closing thought
Elasticsearch is an excellent tool. It just wasn’t the right one for a problem that required trust, not speed.
Experience has taught me that architectural maturity isn’t about knowing more technologies—it’s about knowing when not to use them. That lesson only comes from shipping, breaking things, and owning the consequences.