Claude Code Curl Alternative: When Websites Block Your Requests
Source: Dev.to
Source: Dev.to
Claude Code Curl Alternative: When Websites Block Your Requests
You’ve got Claude Code doing research or pulling data—and it leans on curl to fetch pages. It works like a charm at first, then you hit a site that slams the door: 403 Forbidden, access denied, or just a blank screen.
“Fine, I’ll spin up Playwright locally.”
Real browser, real JavaScript—what could go wrong?
Turns out: plenty. Local Playwright adds serious overhead and often runs into the same anti‑bot walls. There’s a smoother path.
Why curl Fails
| Problem | What Happens |
|---|---|
| Missing or suspicious headers | curl sends minimal headers. Sites expect User‑Agent, Accept‑Language, Accept‑Encoding, etc. Without them you look like a bot. |
| No JavaScript execution | Many sites deliver a skeleton HTML and load the real content via JS. curl only grabs the shell → empty pages. |
| IP‑based rate limiting | One IP, lots of requests. Whether it’s your machine or Claude’s backend, you burn through limits fast. |
| Bot detection | TLS fingerprints, header order, timing, behavior. curl’s signature is easy to read. |
| Result | 403 Forbidden, 429 Too Many Requests, 503 Service Unavailable, or blank pages → Claude Code has nothing to work with. |
Why Local Playwright Isn’t a Silver Bullet
| Drawback | Impact |
|---|---|
| Heavy dependencies | Chromium/Firefox/WebKit → hundreds of MB to install & keep updated. |
| Memory & CPU | Each headless instance chews RAM & CPU. Parallel crawls become expensive fast. |
| Environment headaches | Different OSes, Node versions, Playwright versions → “Works on my machine” becomes a new catch‑phrase. |
| Still blocked by advanced bot detection | • Cloudflare challenges (Turnstile, JS challenges, “Checking your browser” screens) • Headless detection ( navigator.webdriver, missing plugins, odd resolutions)• Behavioral analysis (no mouse moves, unnatural scroll speed) • Proxy/VPN detection (datacenter IPs flagged) |
| Result | Same 403s, Cloudflare blocks, captchas – but now you’re lugging a heavier stack. |
AnyCrawl – Crawl for AI
AnyCrawl is a cloud‑scraping API built for AI agents and LLMs. It bypasses typical limits (including Cloudflare challenges) without requiring a local browser. The service returns LLM‑optimized Markdown, ready for Claude, ChatGPT, and other models. When curl or a local Playwright instance hits a wall, AnyCrawl is the drop‑in replacement.
| Feature | Curl | AnyCrawl |
|---|---|---|
| Headers | Minimal → easy to fingerprint | Full browser‑like headers & request patterns |
| JavaScript | None | Cheerio, Playwright, and Puppeteer engines in cloud |
| IP handling | Single IP → easily rate‑limited | Rotating proxy support to distribute traffic |
| Anti‑bot | Blocked by Cloudflare, 403, etc. | Bypasses Cloudflare and manages anti‑detection |
How AnyCrawl Works
- Send a URL – AnyCrawl fetches it (optionally executing JavaScript).
- Receive clean Markdown (or raw HTML) optimized for AI consumption.
- No local browser required.
- No proxy configuration needed.
- No ongoing maintenance.
Using AnyCrawl with Claude Code
1. Install the AnyCrawl CLI
# Global install
npm install -g anycrawl-cli
# Or one‑off use
npx -y anycrawl-cli init2. Authenticate
# Get your API key at https://anycrawl.dev/dashboard
anycrawl login --api-key <YOUR_API_KEY>3. Install the Claude Code skill
anycrawl setup skillsThe skill teaches Claude Code when and how to use AnyCrawl for scraping, search, mapping, and crawling—automatically swapping in AnyCrawl whenever curl gets blocked.
4. (Optional) MCP Integration
claude mcp add --transport stdio anycrawl \
-e ANYCRAWL_API_KEY=YOUR_API_KEY \
-- npx -y anycrawl-mcp@latest- Replace
YOUR_API_KEYwith the key from your dashboard. - You can also run
anycrawl setup mcpafter installing the CLI.
5. Call the API Directly (scripts or other AIs)
curl -X POST "https://api.anycrawl.dev/v1/scrape" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/page-that-blocks-curl",
"engine": "playwright"
}'The response includes markdown (or html) optimized for AI—ready for Claude or any LLM.
Pricing & Credits
| Plan | Free credits | Paid start | What credits cover |
|---|---|---|---|
| Free | 1,500 credits / month (no card required) | — | - Cheerio (static HTML) – lowest cost - Playwright (JS‑heavy) – moderate cost - Puppeteer – similar to Playwright for Chrome‑specific needs |
| Hobby | — | 2,500 credits / month | Higher volume, priority support |
| Enterprise | — | Custom | Unlimited scaling, dedicated proxies, SLA |
Typical research, docs lookup, and small‑scale extraction (dozens to a few hundred pages/month) fit comfortably within the free tier.
Choosing the Right Tool
| Scenario | Best Tool |
|---|---|
| Simple public pages, no anti‑bot | Claude Code curl |
| JavaScript‑heavy but not protected | Local Playwright (if you’re fine with the setup) |
| Protected sites (Cloudflare, 403/429, captchas) | AnyCrawl |
| High volume, many domains | AnyCrawl (cloud, proxies, concurrency) |
| Claude Code integration | AnyCrawl CLI + skill (anycrawl setup skills) |
Get Started
Sign up at your preferred platform.
Promo code:
STARTUPYGrab your API key from the dashboard.
Install the CLI & skill:
anycrawl setup skills # see the CLI docs for detailsOr call the Scrape API directly from your own scripts.
No credit card needed for the free tier. Try scraping a URL that blocks
curl—the difference shows up right away.
TL;DR
curl– perfect for open, static pages.- Local Playwright – adds complexity and often still fails on protected sites.
- AnyCrawl – cloud‑based, bypasses Cloudflare & anti‑bot measures, uses rotating proxies, handles concurrency, and returns AI‑ready Markdown.
Use AnyCrawl as a drop‑in replacement when curl gets blocked, and let Claude Code focus on what it does best: reasoning over clean, structured data.
AnyCrawl runs headless browsers in the cloud, bypasses anti‑bot protections, and returns LLM‑ready Markdown. For daily use, 1,500 free credits cover most research and extraction—without running browsers locally or wrestling anti‑bot systems yourself.