Scrapy

Say you want every product and price from an online shop, or every article on a news site. Copying them by hand would take weeks. Scrapy does it for you: you describe which links to follow and which details to pick out of each page, and it visits the pages and saves what it finds as tidy data, like a spreadsheet file. It's a Python library, so you write a small amount of code to tell it what to do.

What it can do

• Get through big sites quickly. It fetches many pages at once and slows itself down automatically so it doesn't overload the site.
• Pick out exactly what you need. You point at the parts of a page you want, such as a title, a price or a date, and it pulls them out of every page the same way.
• Tidy up as it goes. Clean, check and filter each result before it's saved, and skip the duplicates.
• Save it how you like. Results come out as common file types (JSON, CSV or XML), or go straight into your own database.
• Grow with add-ons. Extras for logging in, retrying failed pages, working through other networks and much more, most of them free.

Where it fits

Scrapy is at its best on large sites where the information is already in the page when it arrives, like most shops, directories and news sites. It turns a whole site into rows of data that you can then analyse, search or hand to an AI. Some modern sites only show their content after they load in a browser. For those, an add-on called scrapy-playwright lets Scrapy open pages in a real browser, or you can use Crawl4AI or Crawlee instead.

Good to know

• You need to be comfortable with some Python. It's a toolkit for building your own collector, not an app you click through.
• Out of the box it reads pages as they arrive and doesn't run them in a browser, so content that appears later won't be there without the add-on above.
• It gives you data, not clean text for AI. If you want pages turned into readable text for a chatbot, Crawl4AI does that for you.
• Crawl politely: new projects follow each site's rules (its robots.txt file) by default, so keep that on, and respect sites' terms of use.
• Licence: BSD 3-Clause. You can use it for free, including in paid products; just keep the copyright notice.