Data & Search updates

Latest

Crawl4AI
84.2k GitHub stars
Reads websites for you and turns them into clean text or tidy data your AI can use, even pages that only load in a browser.
- Web scraping & crawling
- RAG & agents
- Apache 2.0 with attribution clause
Apache Superset
74.9k GitHub stars
Apache Superset is a Data Visualization and Data Exploration Platform
- Business intelligence
- Apache 2.0
Scrapy
64.5k GitHub stars
A fast, long-trusted Python tool for collecting information from large websites and saving it as tidy data you can use.
- Web scraping & crawling
- Command line tools
- BSD 3-Clause