Crawl4AI, Scrapy or Crawlee? How to pick the right web crawler

Sooner or later, most projects need something from the web. Maybe you want a chatbot that knows your help pages, a spreadsheet of prices from a few shops, or a list of every event in your city. Copying it by hand works for ten pages. For a thousand, you want a crawler: a program that visits pages for you, follows their links and saves what it finds.

Three free, open source crawlers come up again and again: Crawl4AI, Scrapy and Crawlee. They all "read websites", so it's easy to think they're the same. They aren't. Each one was built for a different job, and picking the right one saves you a lot of time.

The short version

Crawl4AI

Best forTurning websites into text for an AI

Opens pages like a real browser and gives back clean, readable text, ready for a chatbot or AI assistant. The quickest way to let your AI read the web.

Language

Python

Opens a browser

On every page

Computer power

High

Scrapy

Best forCollecting lots of data, fast

Reads pages directly without a browser, so it's the fastest and lightest of the three. It's also the oldest and most proven. Best for big sites where the information is already in the page.

Language

Python

Opens a browser

Only with an add-on

Computer power

Low

Crawlee

Best forCrawls that have to keep going

Reads simple pages directly and opens a browser only when a page needs one. Built to survive errors, crashes and sites that push back, so it suits long runs.

Language

JavaScript or Python

Opens a browser

Only where needed

Computer power

Low to high

Now let's look at what actually makes them different.

1. What you get back

This is the biggest difference, so start here.

• Crawl4AI gives you text. It strips away menus, ads and pop-ups and hands you the article or page content as clean Markdown (plain text with headings and lists). That's exactly what AI tools want. If your goal is "let my AI answer questions from these pages", you're mostly done once Crawl4AI has run.
• Scrapy gives you data. You tell it which bits of a page you care about, like the title, the price and the date, and it pulls those out of every page into rows. The result is a spreadsheet-style file (CSV or JSON) or rows in your database. It won't write you a readable summary of the page; that isn't its job.
• Crawlee gives you whatever you ask for. It takes care of visiting the pages reliably and hands each one to your code. You decide what to keep: data, text, screenshots or whole pages.

2. Does it need a browser?

Some websites send you the whole page straight away. Others send an almost empty page and fill it in afterwards with JavaScript, the way a lot of modern shops and apps work. To read those, a crawler has to open the page in a real (hidden) browser and wait.

• Crawl4AI always uses a browser. It sees what a person would see, which is why it handles tricky sites well, but it also makes every page slower and heavier.
• Scrapy doesn't use a browser at all by default. That's what makes it so fast. For pages that need one, you can add the scrapy-playwright add-on.
• Crawlee lets you choose, and you can mix: a fast crawler for simple pages and a browser crawler for the rest, with the same code. Newer versions can even try a page without a browser first and switch only when they have to.

Quick test: open the page, right-click and choose "View page source". If you can find the text you want in there, you don't need a browser. If you can't, you do.

3. Speed and cost

Opening a browser for every page takes a lot of memory and time. Reading a page directly takes very little.

So for a few hundred pages, it hardly matters which you pick. For hundreds of thousands, it matters a lot: Scrapy or Crawlee's non-browser mode can do on one laptop what would need several machines with a browser-based crawler. If you pay for a server by the hour, this is where the difference shows up on your bill.

4. When things go wrong

On a long crawl, things always go wrong. Pages time out, a site starts refusing you, or your computer goes to sleep.

• Crawlee is built for exactly this. It keeps a to-do list of pages on disk, tries again when a page fails, and carries on where it stopped after a crash instead of starting over. It can also spread its visits across different networks and look more like a normal visitor, so long runs are less likely to be cut off.
• Scrapy retries failed pages and can pause and resume a crawl if you turn that on. Its add-ons cover most other needs.
• Crawl4AI retries and can work through a list of pages in parallel, but it's happiest with focused jobs, like "read these 200 help pages", rather than crawling a whole site for days.

5. How much setup

• Crawl4AI is the quickest to get going. Install it (or run it with Docker) and you can turn a page into text with a few lines of code, or through its built-in server.
• Scrapy asks you to learn its way of doing things: a project folder, "spiders" that describe a site, and settings. It's well documented, and once it clicks, it's very productive.
• Crawlee sits in between. It gives you a ready-made project to start from, and the docs are full of examples.

All three are toolkits for programmers, not apps you click through. But you don't have to be one: an AI coding assistant can write the code for you, as the prompts below show.

6. Licences

All three let you use them for free, including in paid products:

• Scrapy: BSD 3-Clause. Keep the copyright notice and you're fine.
• Crawlee: Apache 2.0.
• Crawl4AI: Apache 2.0 with one extra condition. If you use it publicly, you need to credit Crawl4AI the way its licence asks. Check the licence box on its page for the exact wording.

Try each one yourself

The easiest way to feel the difference is to give all three the same job. The two sites below are practice sites made for learning to scrape, so you're welcome to crawl them. The second one only shows its quotes after it loads in a browser, which is a nice test.

Paste each prompt into your AI coding assistant (like Claude Code, Cursor or Codex) in an empty folder. It installs what's needed, writes the code, runs it and shows you the result.

Crawl4AI: a page as clean text

Using Crawl4AI in Python, read https://quotes.toscrape.com/js/ and save the page as clean Markdown in a file called quotes.md. Set everything up for me, run it, and show me the first few lines of the file. I'm a beginner, so explain each step in plain words.

Scrapy: a whole site as a spreadsheet

Using Scrapy, collect every book on https://books.toscrape.com (all 50 pages): title, price, star rating and whether it's in stock. Save it all to books.csv. Keep robots.txt on and crawl politely. Set it up, run it and tell me how many books you found and how long it took. Explain each step in plain words.

Crawlee: a mix of both

Using Crawlee for Python, collect every quote, author and tag from https://quotes.toscrape.com/js/, following the "Next" links to the end. That site needs a browser, so use Crawlee's Playwright crawler. Save the results as JSON. Set it up, run it and tell me how many quotes you found. Explain each step in plain words.

You'll notice that Scrapy gets through its 50 pages the fastest, that Crawl4AI's file reads like a tidy document, and that Crawlee handles the browser-only site with a crawler that would keep going just as well on 50,000 pages.

Use them together

You don't have to pick only one. A common setup for AI projects is to let Scrapy or Crawlee find and fetch the pages, since they're great at getting around big sites, and then pass the pages that matter to Crawl4AI to turn them into clean text for your AI. If you're building something like that, our guide to building a read-later app with Crawl4AI and Supabase shows the second half in action.

Be a good visitor

A crawler is a guest on someone else's website. Whichever tool you pick:

• Follow the site's rules in its robots.txt file, and read its terms of use.
• Go slowly. A few pages at a time is plenty, and all three tools can limit their pace.
• Only collect what you need, and be careful with anything personal.
• If a site offers an official download or API, use that instead.