Hmm, where were we?

Source: Dev.to

Introduction

I received a comment on my recent entry, “AI, the New UI, Not the New API”. The comment reads:

We are already at the point where supermarkets is putting the responsibility for their AI replacing products back on the customer who ordered something… Now multiply that across all domains… And then imagine the cost!

This use‑case highlights a real‑world symptom of a shortcut, an architectural problem: What happens when you let a probabilistic agent execute concrete system changes without strict constraints?

Tacit Knowledge

In an earlier post, I called humans “entropy reducers.” Just as AI is a probabilistic reducer, humans are too. The difference is that humans, knowingly or unknowingly, do not expose much of the context they possess. AI simply does not have access to that context to make similar decisions. This touches on a highly debated topic in AI research: tacit knowledge and the hidden state.

The Power of the Hidden State

In the grocery substitution use‑case, when a human makes a decision, they draw on learned experience:

“The shop is out of whole milk. I will pick toned milk, but never soy milk because my partner hates the taste.”

That preference is a hidden state. The human has access to it, but the grocery list lacks it. The AI making the substitution in the supermarket app only has the explicit context window it was given (the user’s cart, the store inventory, and a generic similarity‑matching algorithm). It fails because it lacks the lifetime of multimodal training data the human carries around. The AI will catch up eventually once the preference is registered; until then, the user gets soy milk.

The Context Bottleneck

This creates a massive bottleneck in system design. We want AI to handle ambiguous, probabilistic tasks, but we do not yet have a frictionless way to transfer our vast human context into the machine’s context window.

When humans unknowingly fail to expose this context, the AI falls back on generic probabilities. It doesn’t make a “bad” decision; it makes the most mathematically likely decision based on a severely restricted dataset.

Humans: The Ultimate Entropy Reducers, Really?

Reflecting on my own entries, it dawns on me that humans are actually more like context anchors. We ground our probabilistic decisions in a strict, personalized reality. When an AI hallucinates a bad grocery substitution, the system relies on the human to step in, apply hidden context, and reject the error, thereby collapsing the probability wave into a definitive, deterministic outcome.

But Then There Is Context Bloat

Can we fix the user‑preference problem? Yes, we can introduce corrective micro‑prompts to nudge the AI’s decision‑making. The problem is that corrective prompts are post‑facto, and in a stateless conversation they are essentially useless. Appending them all just makes the context bloated.

Enter RAG (Retrieval‑Augmented Generation)

You don’t want to overwhelm the AI with context bloat. The simple alternative seems to be: train it again with the additional context. It will work as expected—but at what cost? Retraining an AI is a notoriously expensive affair, a FinOps nightmare.

RAG helps by keeping the model frozen while allowing it to query a database of user preferences and retrieve only the meaningful bits of context, injecting them just‑in‑time for the AI to make a better decision.

Tying It Together

In fact, we do not even need an AI example to illustrate the core issue.

The Problem: Mental Model Mismatch

The classic up/down buttons in a lobby suffer from a UX failure: they require the user to translate their goal into a rigid system command, often leading to the “wrong instruction.” Are those buttons for registering an intent, or for commanding the elevator to a specific floor? There are no hints, and the UX simply fails to convey the point.

When a user on Floor 3 wants to go to Floor 10, they are supposed to press UP (their intended direction of travel). However, if they see the elevator is currently on Floor 15, many will press DOWN, thinking they need to “call the elevator down to me.” The system’s API expects the user’s desired trajectory, but the user is trying to issue a mechanical command to the machine. This low‑context interaction creates errors.

The Solution: Destination Dispatch

With modern technology and sufficient resources, it is now possible to add individual buttons for all floors directly in the lobby. The UX simplifies to “Which floor?”—no more guessing what the buttons mean.

This solution highlights that UX still matters. Capturing a sanitized intent upfront is exactly what allows an AI to make better, safer decisions.