Architecture, Deployment & Observability - The Part Nobody Warns You About
Source: Dev.to
Where the problem really starts
Most people say the trouble begins at requirement gathering – understanding goals, vision, and stakeholder expectations. Those things matter, absolutely.
But honestly? That’s not where things fall apart.
The actual mess starts at technical planning.
- You architect a solution, then try to execute that architecture on real infrastructure that never behaves the way your diagram assumed.
- Once the foundation has cracks, no amount of clean code or good intentions can hide them.
Every decision cascades
- Apache Airflow today → two years later you’re debugging consumer lag at 2 AM.
- DAGs for speed → scaling becomes a six‑month refactoring nightmare.
- Managed cloud service to save time → you’re locked into that provider’s pricing forever.
There is no perfect architecture. The job is picking the right trade‑off for the right context and being honest about what you’re giving up.
| Trade‑off | What to consider |
|---|---|
| Consistency vs. availability | Pick a side, document why. |
| Stateless vs. stateful | Each has infra implications your ops team will live with long after you move on. |
| Managed cloud vs. self‑hosted | A cost‑vs‑control conversation that needs actual numbers, not vibes. |
| Microservices vs. monolith vs. modular monolith | “It depends” is fine, but the dependence must be explicit. |
The engineers who become architects aren’t the ones who know every pattern. They’re the ones who know which pattern to avoid in a given situation. That’s the real experience.
Communicating constraints to the client
- You’ve done the analysis. You know the limitations, why the proposed approach won’t scale, and the real infra constraints.
- Now you have to explain it to someone who paid good money and expects more than the resources can deliver.
If you keep explaining scarce resources in technical terms, the bid is off and you’re valued low – not because you’re wrong, but because you failed to translate the constraint into something the client actually understands.
Lead with outcomes
- “This approach handles a 10× traffic spike without manual intervention.”
- vs. “We’re implementing HPA with custom metrics on Kubernetes.”
Both mean the same thing; only the first keeps the client in the room.
The real game is:
- Make the client understand the boundaries.
- Show them the best possible outcome within those boundaries.
That takes experience, technical knowledge, and a lot of patience.
Architecture on paper vs. architecture in production
Beautiful designs fall apart the first time they hit real network latency, real disk I/O, and real users doing things nobody anticipated in the design session.
Kubernetes – powerful and unforgiving
If you don’t understand what you’re deploying into…
- Node affinity
- Resource requests vs. limits
- Pod disruption budgets
Kubernetes will punish you with fancy errors and silent failures at the worst possible time.
Cloud strategies
Companies are moving in two directions:
| Strategy | Benefits | Drawbacks |
|---|---|---|
| Single‑cloud | Depth, tighter integrations, simpler operational model, better managed‑service compatibility. | Vendor lock‑in, reliance on one roadmap and pricing history. |
| Multi‑cloud | Resilience, leverage (no single‑provider outage takes you down, no pricing lock‑in). | Complexity – managing abstractions across multiple APIs, IAM models, and networking topologies. |
Only a clever, well‑budgeted team survives cloud optimally.
- The cloud is pay‑as‑you‑go, but waste is also pay‑as‑you‑go.
- Waste compounds faster than most teams realize.
AI model hosting – the hidden cost
People used to worry about hosting AI models locally (workload, stability, scalability, performance, latency, accessibility, privacy). So everyone moved to the cloud.
Now the problem is: cloud is pay‑as‑you‑go and the charges are heavier than expected.
- GPU‑backed compute is expensive.
- Cold‑start latency on inference endpoints differs from a stateless REST API.
- Token‑by‑token generation means time‑to‑first‑token and total generation time are completely different signals with different infra implications.
The API call looks cheap. The infrastructure required to make that API call reliable, fast, and cost‑efficient at scale is where the real engineering work hides.
Observability – not just monitoring
Observability ≠ Monitoring
- Monitoring tells you something is wrong (the alarm).
- Observability is the ability to ask arbitrary questions about your system’s internal state based on the signals it produces. It’s how you go from “something is broken” to “here is exactly why, and here is exactly where.”
The three pillars
| Pillar | Purpose | Typical use |
|---|---|---|
| Metrics | What is happening right now. | SLA dashboards, capacity planning, early‑warning systems. |
| Logs | What happened. | Debugging; can be expensive and noisy at scale if you’re not intentional about log levels and sampling. |
| Traces | How it happened. | Full request journey across distributed services. In a microservices world, traces are non‑negotiable. Without them, debugging a latency spike across four services and two external APIs is just educated guesswork. |
Common mistake: treating observability as an afterthought. By the time you bolt it on, instrumentation is inconsistent, naming conventions are all over the place, and you’re collecting a mess of data that’s hard to use.
TL;DR
- The real trouble starts at technical planning, not requirements.
- Every architectural decision ripples through cost, complexity, and operability.
- Communicate outcomes, not jargon, to keep stakeholders on board.
- Expect a gap between design and production; invest early in solid Kubernetes and cloud practices.
- For AI workloads, factor in GPU cost, cold‑start latency, and token‑level performance.
- Build observability from day one—metrics, logs, and traces—to turn “something’s broken” into actionable insight.
Observability for AI‑Powered Systems
Traditional APM tooling wasn’t built for LLM workloads. Latency behaves differently. But beyond infrastructure metrics, AI systems need semantic observability.
- Did the model return something useful?
- Was the retrieved context in the RAG pipeline actually relevant?
- Is the prompt structure degrading as edge cases accumulate over time?
CPU utilization and memory graphs can’t answer those questions. You need:
- Eval pipelines
- Response‑quality sampling
- Feedback loops embedded into the product itself
That’s a different layer of observability, and most teams aren’t thinking about it yet.
In cloud‑native environments, an unexpected cost spike is often the first indicator of a misconfiguration or a runaway process. Engineers who treat FinOps as someone else’s problem eventually end up in an awkward conversation with leadership trying to explain why infra costs tripled.
- Tagging resources
- Attributing costs to services and teams
- Anomaly alerts on spend
These are observability tasks and belong in the same operational posture as your Prometheus dashboards.
The cloud is more capable than ever. AI capabilities are an API call away. Kubernetes lets you orchestrate globally. The tooling exists and it’s genuinely impressive—but tooling is not a substitute for craft.
The engineer who can:
- Design a system that survives its own success
- Deploy it reproducibly and observably
- Instrument it to give genuine insight into its actual behavior
is rare. That combination is what actually moves organizations forward.
The gap between a system that technically works and a system that is production‑ready, cost‑efficient, observable, and maintainable is not small. In most projects, that gap is the majority of the actual engineering effort.
Requirement gathering gave you a direction. Architecture, deployment, and observability are the journey.
Anyone can deploy a service. Fewer can architect one that lasts. Fewer still can tell you, at any moment, exactly how that service is behaving and why.
That’s the skill set. That’s the discipline. And as AI workloads and cloud‑native systems keep evolving, the engineers who invest in all three—not just the one they find most interesting—are the ones building the infrastructure the next decade runs on.
The cloud isn’t magic. Kubernetes isn’t magic. AI isn’t magic. The craft is understanding deeply enough to make it look that way.
~ Ranga Bashyam