I Made Two AIs Review My robots.txt — Here’s What They Taught Me About SEO and Over-Optimization
Source: Dev.to
The Setup
I’m currently building SemesterExam.com, a platform providing semester‑wise engineering notes.
Before deployment, I finalized my robots.txt:
- Blocked private routes (
/api,/admin,/auth) - Prevented crawling of tracking parameters (
utm,fbclid, etc.) - Allowed
Googlebot-Image - Blocked AI crawlers
- Added proper sitemap reference
Everything looked production‑ready, but I asked two AIs to review it.
What Claude Said
“This is superior. Production‑ready. No changes needed.”
Confident. Direct. Clear.
What ChatGPT Said
“It’s good. But those extra rules don’t boost SEO. They just improve clarity.”
More nuanced. Slightly critical. Still positive.
Who Was Right?
Both. And that’s where it got interesting.
The Technical Reality
- Separate
Googlebotrules don’t magically improve rankings. - If
User-agent: *already allows something, Google follows it. - Extra bot‑specific blocks improve clarity, not SEO power.
- Clean structure ≠ better performance.
In other words, my configuration was production‑ready, but it wasn’t “superior.” It was simply well‑structured, and that distinction matters.
The Bigger Lesson
Most developers stop at:
“AI said it’s good.”
Real engineering starts at:
“Why is it good?”
AI validation is useful. AI verification is powerful. There’s a difference between:
- Seeking reassurance
- Seeking understanding
That 15‑minute robots.txt debate taught me more than hours of passive SEO tutorials.
Why This Matters for Developers
In 2026, AI can:
- Review code
- Suggest optimizations
- Detect improvements
But it can also:
- Overstate benefits
- Use strong language like “superior”
- Make something sound more impactful than it is
If you don’t question it, you inherit its confidence — not its reasoning.
Final Thought
Building in public has forced me to slow down and verify things properly. Sometimes the best optimization isn’t adding more rules; it’s understanding the ones you already have.
What’s a time you caught AI — or yourself — overstating something?
