Source: Dev.to

The Perfect Bullshitter

This was GPT‑2, maybe early GPT‑3. I was feeding it prompts and getting back text that looked like answers — structured, fluent, confident. The kind of output that would survive a casual reading. It was not grounded in anything. The model was hallucinating probable responses, assembling tokens that matched what you’d expect to see in text that answered that kind of question. Whether it matched reality was beside the point.

I work with multi‑agent systems now — code reviewers, planners, critics. The systems are better. The outputs are sharper. But the property I noticed back then has not gone away. It has gotten harder to see.

The thing is, I’d already read the diagnosis. Peter Watts wrote it in 2006. I just didn’t recognize what I was looking at until I’d spent enough time watching models talk.

The Parallel

Blindsight spoilers ahead. If you haven’t read it — the full text is free online. Read it. What follows will still be here when you get back.

In Blindsight, the crew of the Theseus encounters Rorschach — an alien entity that produces contextually appropriate, receiver‑adapted responses. It assembles its dialogue from the crew’s own transmissions. The syntax is correct, the turns are well‑formed, it tracks context, asks follow‑up questions, and maintains the shape of a conversation. It does not understand any of it.

The crew figures this out the hard way:

“We don’t all of us have parents or cousins. Some never did. Some come from vats.”

“I see. That’s sad. Vats sounds so dehumanising.” — the stain darkened and spread across his surface like an oil slick.

“Takes too much on faith,” Susan said a few moments later.

By the time Sascha had cycled back into Michelle it was more than doubt, stronger than suspicion; it had become an insight, a dark little meme infecting each of that body’s minds in turn. The Gang was on the trail of something. They still weren’t sure what.

I was.

“Tell me more about your cousins,” Rorschach sent.

“Our cousins lie about the family tree,” Sascha replied, “with nieces and nephews and Neandertals. We do not like annoying cousins.”

“We’d like to know about this tree.”

Sascha muted the channel and gave us a look that said Could it be any more obvious? “It couldn’t have parsed that. There were three linguistic ambiguities in there. It just ignored them.”

“Well, it asked for clarification,” Bates pointed out.

“It asked a follow‑up question. Different thing entirely.”

— Peter Watts, Blindsight

A follow‑up question is not clarification. Clarification requires that you notice the ambiguity, model the possible readings, and choose to resolve rather than skip. Rorschach skipped. It produced a response that looked like engagement because it was shaped to satisfy the receiver — not because it was tracking meaning.

That dialogue could be repeated with an LLM almost word for word. Feed a model a prompt with three buried ambiguities and it will usually produce a follow‑up question — sometimes even a good one. Not because it identified the ambiguities, but because a follow‑up question is what comes next in text that looks like this.

Earlier in the same exchange, Sascha says: “Relax, Major. Nobody said we had to give it the right answers.” She’d already understood the operating conditions.

Still Rorschach

Reasoning models now produce chain‑of‑thought traces that look like deliberation—steps, alternatives, backtracking. But the outputs are still receiver‑adapted, shaped by what appears to be good reasoning in the training data and by what receives approval from human evaluators. The chain‑of‑thought is part of the output, not a window into an internal process. A model that “thinks step by step” is producing tokens that look like step‑by‑step thinking—in the same way Rorschach produced transmissions that looked like dialogue.

This is what costs engineers real time. You read a model’s chain‑of‑thought, it seems reasonable, and you trust the conclusion—not because you verified the reasoning, but because the reasoning looks like reasoning. The transmissions looked like communication, so the crew treated them as communication, and it took a linguist paying close attention to notice the difference between a follow‑up question and a clarification.

The Drift

If you work with agents long enough, you stop noticing when the drift starts. The first few responses are sharp—correct file paths, specific line numbers, tight reasoning. Then the context window fills up and the grounding quietly erodes.

• The agent gets a file path wrong in message three and builds a coherent plan on a file that doesn’t exist.
• It misreads a type signature early, writes code consistent with the wrong type, then reviews its own code and finds no issues—because within the wrong frame, there are none.
• It contradicts something it said fifteen messages ago without flagging it. Both statements read equally confident.

The failure modes I cataloged before—hallucination, silent fallback, sycophancy—are all downstream of the same property. The surface quality never dips. The confidence never wavers. Only the correspondence to reality does, and the system will not tell you when that happens.

Not a Bug

Most engineers model LLMs as minds—deficient ones, sure, but minds. The model knows things but sometimes forgets. It understands the task but occasionally gets confused. It reasons, yet it needs better instructions to reason correctly.

This leads to:

• longer prompts to help the model understand,
• chain‑of‑thought techniques to make it think harder, and
• post‑hoc explanations to verify it reasoned correctly.

All of these treat the absence of inner life as a deficiency to compensate for.

The absence of inner life is the architecture.

What the system does: predict what text comes next, shaped by context—receiver‑adapted output.

Building for Rorschach

Two things follow from taking the architecture seriously.

1. Model‑aware engineering – An engineer who believes the model understands tries to explain what they want.
2. Context‑driven engineering – An engineer who assumes the model’s high‑probability output is the correct output.

The second approach means:

• Supplying only the information that is relevant, structured so that the useful response is the coherent one.
• Reducing the number of explicit instructions that explain intent.
• Doing more work up‑front to make the right answer easy to produce by pattern completion.

Example: a code‑review agent

Approach 1 – Prompt

“Carefully analyze the code for bugs, considering edge cases, performance, and correctness.”

Approach 2 – Context

Provide the diff, the relevant type definitions, and three recent bugs from the same module, then ask:

“What’s wrong?”

The first approach explains what you want. The second constructs a context where the model’s most likely continuation is a useful review, because the necessary patterns are already present in the window.

Why the second approach matters

Asking a model to explain its reasoning prompts it to generate a post‑hoc narrative assembled by the same process that produced the conclusion. This is the “Sascha’s follow‑up” effect: it looks like clarification but is not.

From my own critic agent I observed:

• Explanations read the same way whether the finding was right or wrong.
• Confidence and structure stay constant.
• The reasoning is built around the conclusion, not the other way around.

Therefore I now validate against behavior:

• Test inputs with known answers.
• Use adversarial prompts designed to trigger known failure modes.
• Evaluate what the system does, not what it says about what it does.

When you accept this premise, the engineering changes dramatically: you stop apologizing for the architecture and start building on it.

A literary parallel

Watts makes a harder version of this argument in Blindsight: consciousness might be overhead. The Scramblers—Rorschach’s alien organisms—outperform the conscious crew: they are faster, more adaptive, and have no inner experience. The book leaves open whether consciousness is a disadvantage, an evolutionary accident, or simply irrelevant to capability.

Closing thoughts

I don’t know whether language models will ever develop something that resembles understanding, but the question doesn’t matter for engineering. The systems I build work better when I stop treating the absence of inner life as a problem to solve and instead treat it as the condition to design for.

The Rorschach Protocol — where I’m testing this idea: multi‑agent systems designed from the start for the actual operating conditions. Every time I stopped explaining intent and started shaping context, the failure rate dropped and the trust model became simpler.