I built 'git log' for any website — track changes with diffs and AI summaries

Source: Dev.to

Introduction

Websites change all the time—pricing pages, docs, terms of service. I wanted a simple CLI to tell me what changed, like git log but for any URL, so I built crawldiff.

Installation

pip install crawldiff

Snapshot a site

crawldiff crawl https://stripe.com/pricing

See what changed later

crawldiff diff https://stripe.com/pricing --since 7d

Features

• Git‑style diffs – colored unified diffs displayed in the terminal.
• AI summaries – generate concise change summaries using Claude, GPT, or Cloudflare Workers AI (e.g., “Pricing increased from $25 to $30, new Enterprise tier added”).
• Watch mode – continuously monitor a URL (e.g., crawldiff watch https://competitor.com --every 1h).
• Multiple output formats – terminal, JSON (pipe to jq/Slack), Markdown reports.
• Incremental crawling – only fetches pages that have changed via Cloudflare’s modifiedSince.
• Local storage – all snapshots are stored in SQLite under ~/.crawldiff/.

Tech stack

• Python 3.12
• typer, rich, httpx
• Cloudflare Browser Rendering /crawl API
• difflib for unified diffs
• SQLite for local snapshot storage
• 96 tests, mypy strict, CI on GitHub Actions

Links

• GitHub:
• PyPI:

Feel free to ask any questions!