How I built a scraper that actually works on Cloudflare sites

Source: Dev.to

Why scraping keeps failing

There are three things killing most scrapers right now.

• JavaScript rendering – Many sites serve an empty HTML shell and hydrate via React or Vue. Fetching the URL directly returns only a “ with an ID and nothing else.
• Bot detection – Cloudflare, PerimeterX, DataDome, etc., fingerprint your browser. Missing plugins, wrong screen resolution, or perfectly timed mouse movements trigger blocks. A vanilla Playwright script fails within about 30 seconds.
• IP reputation – Datacenter IPs are flagged before your code even runs. AWS, Hetzner, DigitalOcean, and similar providers are blocked by default on many target sites.

You can fight each of these individually, or you can just not deal with it.

What I built

anybrowse takes a URL and returns clean Markdown. That is the whole API.

pip install anybrowse

from anybrowse import AnybrowseClient

client = AnybrowseClient()
result = client.scrape("https://techcrunch.com")
print(result.markdown)

Under the hood it runs patched Chromium with randomized fingerprints, falls back to residential ISP proxies when the first attempt fails, and uses a Firefox‑based engine (Camoufox) for sites that specifically profile Chrome. CAPTCHA solving is built in via CapSolver.

For AI agents, there is an MCP server that works out of the box with Claude Desktop, Cursor, and Windsurf:

{
  "mcpServers": {
    "anybrowse": {
      "type": "streamable-http",
      "url": "https://anybrowse.dev/mcp"
    }
  }
}

Your agent gets scrape, crawl, search, batch scrape, and structured extraction tools. The search endpoint goes through the Brave Search API so it actually returns results instead of timing out on Google.

Honest numbers

• 90 % success rate on general websites.
• LinkedIn and Twitter remain difficult because they require login for most content.
• Paywalls are a separate problem that scraping does not solve.

The remaining 10 % of failures are mostly aggressive per‑request CAPTCHAs and strict login walls. CapSolver helps but is not magic.

Try it

• 10 free scrapes per day, no signup, no credit card.
• Credit packs start at $5 for 3,000 scrapes if you need more.

anybrowse.dev • GitHub • Docs