Crawlee

A crawler is a program that visits web pages, follows their links and collects what it finds. The basic idea is simple. The trouble starts at scale: pages time out, sites block you, the program crashes halfway through and you have to start again. Crawlee handles all of that for you. You write the part that says what to collect from each page, and Crawlee takes care of visiting the pages reliably.

What it can do

• Fast when it can, a browser when it must. Simple pages are read directly, which is quick and cheap. Pages that only show their content in a browser can be opened in a real one. You can switch between the two without rewriting your crawler.
• Never lose your place. It keeps a list of pages to visit, tries again when one fails, and can carry on after a crash instead of starting from scratch.
• Uses your computer sensibly. It speeds up or slows down depending on how much power your machine has spare.
• Built to run for a long time. It can spread its visits across different networks and behave more like a normal visitor, so long crawls are less likely to be cut off.
• Saves as it goes. Results are stored as they're collected and can be exported as common file types (JSON or CSV).
• JavaScript or Python. There's a version for each, built on the same ideas.

Crawl4AI, Scrapy or Crawlee?

All three are free tools that visit websites and collect what's on them, but each one is built for a different job.

Crawl4AI

Best forTurning websites into text for an AI

Opens every page in a real browser and hands back clean, readable text, ready for a chatbot or AI assistant. The easiest start if you want your AI to read websites.

Language

Python

Opens a browser

On every page

Computer power

High

Scrapy

Best forCollecting lots of data, fast

Reads pages directly without a browser, so it's the quickest and lightest of the three, and the most mature. Best for large sites where the information is already in the page, like shops and directories.

Language

Python

Opens a browser

Only with an add-on

Computer power

Low

CrawleeThis page

Best forCrawls that have to keep going

Reads simple pages directly and switches to a real browser only where it has to. Built to keep going through errors, crashes and sites that push back, so it suits long runs.

Language

JavaScript or Python

Opens a browser

Only where needed

Computer power

Low to high

A quick way to choose: want your AI to read the web? Start with Crawl4AI. Collecting structured data from big, simple sites? Scrapy. Need a crawler that holds up over a long run, or you work in JavaScript? Crawlee. They also combine well: many projects use Crawlee or Scrapy to find and fetch pages, then Crawl4AI to turn them into text for AI.

Good to know

• It's a toolkit for programmers, not an app. You'll write some JavaScript or Python.
• It runs on your own computer or server. Apify also offers a paid cloud to run it for you, but you don't need it.
• Being built to keep going doesn't mean you're allowed to. Follow each site's rules (its robots.txt file) and terms of use, and crawl at a polite pace.
• Licence: Apache 2.0. You can use it for free, including in paid products.