Web Scraping Python Libraries

Search every project in one place

Press / to search. Tap a tag to filter. Click any row for details.

Filtering for

Row number				Tags
1	browser-use Frameworks	107,422	2026-07-27	Frameworks Web Scraping HTTP & Scraping	→
	Make websites accessible for AI agents with easy browser automation.
	browser-use/github.com/browser-use/browser-use /2026-07-27
2	crawl4ai Frameworks	75,714	2026-07-15	Frameworks Web Scraping HTTP & Scraping	→
	An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents.
	unclecode/github.com/unclecode/crawl4ai /2026-07-15
3	scrapy Frameworks	63,518	2026-07-31	Frameworks Web Scraping HTTP & Scraping	→
	A fast high-level screen scraping and web crawling framework.
	scrapy/github.com/scrapy/scrapy /2026-07-31
4	trafilatura Content Extraction	6,379	2026-07-31	Content Extraction Web Scraping HTTP & Scraping	→
	A tool for gathering text and metadata from the web, with built-in content filtering.
	adbar/github.com/adbar/trafilatura /2026-07-31
5	mechanicalsoup Frameworks	4,880	2026-07-29	Frameworks Web Scraping HTTP & Scraping	→
	A Python library for automating interaction with websites.
	MechanicalSoup/github.com/MechanicalSoup/MechanicalSoup /2026-07-29
6	sumy Content Extraction	3,698	2026-07-31	Content Extraction Web Scraping HTTP & Scraping	→
	A module for automatic summarization of text documents and HTML pages.
	miso-belica/github.com/miso-belica/sumy /2026-07-31
7	feedparser Content Extraction	2,406	2026-07-30	Content Extraction Web Scraping HTTP & Scraping	→
	Universal feed parser.
	kurtmckee/github.com/kurtmckee/feedparser /2026-07-30
8	html2text Content Extraction	2,169	2025-10-28	Content Extraction Web Scraping HTTP & Scraping	→
	Convert HTML to Markdown-formatted text.
	Alir3z4/github.com/Alir3z4/html2text /2025-10-28
9	micawber Content Extraction	680	2026-07-19	Content Extraction Web Scraping HTTP & Scraping	→
	A small library for extracting rich content from URLs.
	coleifer/github.com/coleifer/micawber /2026-07-19
10	website-downloader Frameworks	162	2026-07-28	Frameworks Web Scraping HTTP & Scraping	→
	A modern wget --mirror / HTTrack alternative that turns whole websites into browsable offline copies.
	PKHarsimran/github.com/PKHarsimran/website-downloader /2026-07-28
11	crawlberg Frameworks	148	2026-07-31	Frameworks Web Scraping HTTP & Scraping	→
	A high-performance web crawling engine with a Rust core, headless-browser fallback, and built-in robots.txt and sitemap parsing.
	xberg-io/github.com/xberg-io/crawlberg /2026-07-31

Contribute

Know a project that belongs here?

Tell us what it does and why it stands out.

Submit a project Star the repository

Search every project in one place

Search and filter

Results

Know a project that belongs here?