Dissecting the Humanization Pipeline for AI Text: A 6-Step Ablation Study

Source: Dev.to

Ablation Study: Which Transformation Steps Really Matter?

“The scores are good. But what’s actually working?”

In the previous article I built a pipeline to make AI‑generated text feel more human‑like and reported benchmark results of Mean Alignment = 0.945 and Distribution Alignment = 0.864. Those numbers look solid, but they don’t tell us which of the six transformation steps are actually contributing and which are just noise.

What I Did

I performed an ablation (removal) study: each step was disabled one‑by‑one and the pipeline was re‑evaluated on a held‑out test set of 500 samples (80 % / 20 % split).

Below are the results.

Results Table

Key take‑aways:

1. Filler insertion caused the biggest collapse (‑0.323).
2. Long‑sentence splitting and short‑sentence insertion together contributed more than fillers (‑0.376).
3. Self‑correction injection had virtually no impact.

Deep Dive into the Surprises

1️⃣ Filler Insertion – The Biggest Contributor (‑0.323)

Humans regularly use fillers such as “Well,”, “You know,” or “Basically,” (≈ 1 per 6 sentences). AI almost never does, making filler rate the strongest single discriminator.

The False‑Positive Pitfall

My initial implementation flagged any occurrence of the word “like” (\blike\b) as a filler. This mistakenly counted “I like pizza” and inflated the filler rate to > 0.3, leading me to (incorrectly) conclude that humans over‑use fillers.

Fix: Switch to position‑dependent detection:

# NG: Riddled with false positives
FILLER_PATTERNS = [r"\blike\b", r"\bso\b", r"\bwell\b"]

# OK: Detects only filler usage at sentence start + comma
FILLER_START_PATTERNS = [r"^(?:well|so|like)\s*,"]
FILLER_ALWAYS = [r"\byou know\b", r"\bi mean\b", r"\bbasically\b"]

Lesson: In quantitative NLP work, eliminate regex false positives before drawing conclusions.

2️⃣ Self‑Correction Injection – A Failure (‑0.001)

Self‑correction markers (“wait, I mean…”, “sorry, what I meant was…”) barely appear in human business communication (0.19 % / sentence, weight = 0.097). With only 500 samples, the confidence interval is wide ([0.001, 0.004]), burying any effect in noise.

Result: The step was removed from the final pipeline.

3️⃣ Long‑Sentence Splitting & Short‑Sentence Insertion

AI tends to produce uniformly long sentences; humans mix short acknowledgments with longer explanations. Together these steps contribute ‑0.376, surpassing the filler contribution.

Metric Weights (Derived from Discriminative Power)

Effect size d > 0.8 denotes a large effect; thus filler rate, words/sentence, and sentence‑length CV dominate human/AI discrimination.

Limitations & Future Work

• Context‑dependence: The current pipeline injects fillers and hedges at a fixed probability. In reality, filler usage varies by topic (more in casual conversation, less in technical writing). This mismatch caused two metrics to fail the KS test.
• Automated vs. Human Evaluation: The DPO benchmark rewards superficial feature matching (e.g., presence of fillers or typos) but does not guarantee that a human reader feels the text is human‑written. Human evaluation remains essential.
• Sample Size: With only 500 test samples, rare phenomena (e.g., self‑corrections) are hard to assess reliably.

Bottom‑Line Ranking of Steps

Resources

• Code & Data:
• Full research article: (link to next article)

End of cleaned markdown.

Status: Formally published as a preprint

Title: HumanPersonaBase: A Language-Agnostic Framework for Human-Like AI Communication

DOI: 10.5281/zenodo.19273577