The Medical Chatbot That Diagnosed Everyone With Cancer

Source: Dev.to

Day Four: Panic Mode

Customer support was drowning. Users were furious and terrified. People with headaches were told they might have brain tumors. Parents of kids with colds were seeing leukemia in the response. People with coughs were being warned about lung cancer.

I checked the logs and immediately understood what I had done.

What Was Actually Happening

A user said they’d had a headache for three days. The AI responded with a long list of possible causes—tension headaches, migraines, dehydration, eye strain, sinus infections—then added rare but serious possibilities: brain tumors, meningitis, aneurysms. It ended with a warning to consult a doctor immediately.

It did this for everything. Fatigue included leukemia. Coughs included lung cancer. Fever included life‑threatening conditions.

The AI wasn’t technically wrong—those things can happen—but it was functionally dangerous.

Why This Happened

My prompt told the AI to be thorough and cautious: provide comprehensive information, always include serious conditions even if rare, and recommend seeing a doctor when in doubt. The AI followed those instructions perfectly, listing everything, emphasizing seriousness, and escalating every case for every symptom.

The Problem I Created

I built “medical student syndrome” at scale. Medical students learn about many diseases and sometimes believe they have all of them. I turned that experience into a chatbot for everyday users. The AI presented worst‑case scenarios alongside common explanations with equal weight, causing users to fixate on the scariest possibility.

Real User Impact

• A woman with occasional headaches saw “brain tumor” mentioned, didn’t sleep for three days, and went to the ER. Diagnosis: stress.
• A parent saw “leukemia” listed for a child’s fever, rushed to the ER at two in the morning. Diagnosis: viral infection.
• A former smoker with a cough saw “lung cancer,” spiraled, visited the ER, and was diagnosed with seasonal allergies.

Reviews called the app irresponsible and dangerous.

The Legal Scare

In week two, a lawyer contacted us. A user had gone to the ER three times in one week after chatbot interactions. Each time, the AI suggested serious cardiac conditions; the diagnoses were anxiety and panic attacks. The user claimed the chatbot caused the anxiety that sent them to the ER. This revealed a liability, not just bad UX.

My First Failed Fix

I tried restricting serious conditions to severe symptoms. That failed immediately. The AI couldn’t consistently interpret what “severe” meant—some mild symptoms were escalated, and some genuinely concerning cases were underplayed. We were blamed either way.

The Real Solution

I rebuilt the system around likelihood, context, and framing. The AI stopped acting like a medical encyclopedia and started acting like a guide:

• Common conditions are listed first.
• Rare conditions are mentioned only when the symptom pattern, duration, or severity justifies them.
• Single mild symptoms no longer trigger cancer mentions.
• Short durations don’t trigger emergency language.
• Every response clearly explains when and why a doctor visit makes sense.

How Responses Changed

• A three‑day headache now produces reassurance, practical self‑care advice, and clear red‑flag criteria.
• Fatigue produces lifestyle explanations first, with guidance on when to seek medical evaluation.
• Urgent escalation occurs only for true emergencies like chest pain with shortness of breath or stroke symptoms.

The AI stopped shouting and started explaining.

Handling the Hard Cases

• When symptoms are truly concerning, the AI acts decisively—no hedging, no long lists—just clear instructions to seek emergency care.
• When symptoms persist for weeks, it escalates calmly and appropriately.
• When users explicitly express fear, the AI addresses the fear directly, explains probabilities, reassures without dismissing, and acknowledges health anxiety as real.

The Results

• Before the fix, most users reported feeling anxious or scared; ER visits spiked; reviews were brutal; legal risk was real.
• After the fix, panic dropped dramatically, ER visits became rare and appropriate, reviews turned positive, and legal threats disappeared.
• Users described the chatbot as calming, helpful, and reassuring.

What I Learned

• Completeness is not the same as helpfulness.
• Order matters—rare conditions listed alongside common ones feel equally likely.
• Medical information without probability context is irresponsible.
• Healthcare AI needs different safety rules than most other domains.
• Testing only with clinicians is a mistake; real users don’t think in probabilities.

The Principle

Inform without alarming. Guide without frightening. Healthcare AI should reduce anxiety, not create it. The goal isn’t to show everything that could go wrong; it’s to help people make reasonable decisions without panic.

Your Turn

Have you ever built something that was technically correct but practically harmful? How do you balance thoroughness with responsibility in sensitive domains?