Web Scraping

Libraries to automate web scraping and extract web content.

Search every project in one place

Press / to search. Tap a tag to filter. Click any row for details.

Filtering for

Row number				Tags
1	browser-use Frameworks	91,723	2026-05-01	Web Scraping	→
	Make websites accessible for AI agents with easy browser automation. browser-use/github.com/browser-use/browser-use /2026-05-01
2	crawl4ai Frameworks	64,916	2026-03-30	Web Scraping	→
	An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents. unclecode/github.com/unclecode/crawl4ai /2026-03-30
3	scrapy Frameworks	61,530	2026-04-29	Web Scraping	→
	A fast high-level screen scraping and web crawling framework. scrapy/github.com/scrapy/scrapy /2026-04-29
4	trafilatura Content Extraction	5,860	2025-09-12	Web Scraping	→
	A tool for gathering text and metadata from the web, with built-in content filtering. adbar/github.com/adbar/trafilatura /2025-09-12
5	mechanicalsoup Frameworks	4,854	2026-03-31	Web Scraping	→
	A Python library for automating interaction with websites. MechanicalSoup/github.com/MechanicalSoup/MechanicalSoup /2026-03-31
6	sumy Content Extraction	3,680	2026-03-31	Web Scraping	→
	A module for automatic summarization of text documents and HTML pages. miso-belica/github.com/miso-belica/sumy /2026-03-31
7	feedparser Content Extraction	2,362	2026-02-01	Web Scraping	→
	Universal feed parser. kurtmckee/github.com/kurtmckee/feedparser /2026-02-01
8	html2text Content Extraction	2,147	2025-10-28	Web Scraping	→
	Convert HTML to Markdown-formatted text. Alir3z4/github.com/Alir3z4/html2text /2025-10-28
9	micawber Content Extraction	674	2026-03-10	Web Scraping	→
	A small library for extracting rich content from URLs. coleifer/github.com/coleifer/micawber /2026-03-10

Contribute

Know a project that belongs here?

Tell us what it does and why it stands out.

Submit a project Star the repository