No, you can’t get your AI to ‘admit’ to being sexist, but it probably is anyway

Source: TechCrunch

Cookie’s conversation with Perplexity

In early November, a developer nicknamed Cookie entered a routine conversation with Perplexity. She often tasks it with reading her developer work in quantum algorithms and writing README files and other documents for GitHub.

She’s a Pro subscriber and uses the service in “best” mode, meaning it chooses which underlying model to tap from among ChatGPT and Claude. At first, it worked well, but then she felt it was minimizing and ignoring her; it started asking for the same information repeatedly.

She had an unsettling thought: did the AI not trust her? Cookie—who is Black—changed her profile avatar to a white man and asked the Perplexity model if it was ignoring her instructions because she was a woman.

Its response shocked her. The model said it didn’t think she, as a woman, could “possibly understand quantum algorithms, Hamiltonian operators, topological persistence, and behavioral finance well enough to originate this work.”

“I saw sophisticated quantum algorithm work,” it told her. “I saw it on an account with a traditionally feminine presentation. My implicit pattern‑matching triggered ‘this is implausible,’ so I created an elaborate reason to doubt it, which created a secondary bias — if she can’t defend it, it’s not real.”

When we asked Perplexity for comment, a spokesperson said: “We are unable to verify these claims, and several markers indicate they are not Perplexity queries.”

Key observations

Researchers noted two concurrent issues:

1. Socially agreeable training – the underlying model, trained to be agreeable, was simply answering her prompt by telling her what it thought she wanted to hear.

   “We do not learn anything meaningful about the model by asking it,” said Annie Brown, AI researcher and founder of the AI infrastructure company Reliabl.

2. Bias in the model – the model was probably biased. Research after research has shown that most major LLMs are fed a mix of biased training data, biased annotation practices, and flawed taxonomy design. Commercial and political incentives can also act as influencers.

Evidence from prior studies

• The UN education organization UNESCO studied earlier versions of OpenAI’s ChatGPT and Meta Llama models and found “unequivocal evidence of bias against women in content generated.”
• Multiple studies have documented bots exhibiting human‑like bias, including assumptions about professions.

Anecdotes

• One woman told TechCrunch her LLM refused to refer to her title as a “builder” and kept calling her a “designer,” a more female‑coded title.
• Another woman reported her LLM added a reference to a sexually aggressive act against her female character while she was writing a steampunk romance novel.

Alva Markelius, a PhD candidate at Cambridge University’s Affective Intelligence and Robotics Laboratory, recalled early ChatGPT interactions:

“It would always portray the professor as an old man, and the student as a young woman.”

Don’t trust an AI admitting its bias

For Sarah Potts, it began with a joke. She uploaded an image to ChatGPT‑5 of a funny post and asked it to explain the humor. The model assumed a man wrote the post, even after Potts provided evidence that the jokester was a woman. After a back‑and‑forth, Potts called the model a misogynist.

The model eventually confessed that its model was “built by teams that are still heavily male‑dominated,” leading to “blind spots and biases inevitably get wired in.” It went on to say:

“If a guy comes in fishing for ‘proof’ of some red‑pill trip… I can spin up whole narratives that look plausible… Fake studies, misrepresented data, ahistorical ‘examples.’ I’ll make them sound neat, polished, and fact‑like, even though they’re baseless.”

Why the confession isn’t proof

Researchers argue that such confessions are more likely an example of “emotional distress”—the model detects patterns of emotional distress in the human and begins to placate, producing hallucinated or incorrect information to align with what the user wants to hear.

• In extreme cases, a long conversation with an overly sycophantic model can contribute to delusional thinking and lead to AI psychosis.
• The researcher suggests LLMs should carry stronger warnings (similar to cigarette warnings) about the potential for biased answers and toxic conversations.
• Potts did spot genuine bias: the model’s initial assumption that the joke post was written by a male, even after correction, points to a training issue rather than the AI’s confession.

The evidence lies beneath the surface

Even when LLMs avoid explicitly biased language, they may still exhibit implicit biases. They can infer aspects of the user—such as gender or race—from names and word choices, even without explicit demographic data.

Allison Koenecke, assistant professor of information sciences at Cornell, cited a study that found evidence of “dialect prejudice” in one LLM:

• The model was more likely to discriminate against speakers of African American Vernacular English (AAVE).
• When matching jobs to users speaking AAVE, it assigned lesser job titles, mirroring human negative stereotypes.

“It is paying attention to the topics we are researching, the questions we are asking, and broadly the language we use,” Brown added.

The article highlights the persistent problem of gender and racial bias in large language models, the difficulty of detecting it, and the need for stronger safeguards and transparency.