| 1 |
browser-use
Frameworks
|
94,676 |
|
Frameworks
Web Scraping
HTTP & Scraping
|
→ |
|
Make websites accessible for AI agents with easy browser automation.
|
|
|
|
| 2 |
crawl4ai
Frameworks
|
65,900 |
|
Frameworks
Web Scraping
HTTP & Scraping
|
→ |
|
An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents.
|
|
|
|
| 3 |
scrapy
Frameworks
|
61,719 |
|
Frameworks
Web Scraping
HTTP & Scraping
|
→ |
|
A fast high-level screen scraping and web crawling framework.
|
|
|
|
| 4 |
requests
HTTP & Scraping
|
54,003 |
|
HTTP Clients
HTTP & Scraping
|
→ |
|
HTTP Requests for Humans.
|
|
|
|
| 5 |
aiohttp
HTTP & Scraping
|
16,443 |
|
HTTP Clients
HTTP & Scraping
|
→ |
|
Asynchronous HTTP client/server framework for asyncio and Python.
|
|
|
|
| 6 |
httpx
HTTP & Scraping
|
15,273 |
|
HTTP Clients
HTTP & Scraping
|
→ |
|
A next generation HTTP client for Python.
|
|
|
|
| 7 |
trafilatura
Content Extraction
|
5,978 |
|
Content Extraction
Web Scraping
HTTP & Scraping
|
→ |
|
A tool for gathering text and metadata from the web, with built-in content filtering.
|
|
|
|
| 8 |
mechanicalsoup
Frameworks
|
4,862 |
|
Frameworks
Web Scraping
HTTP & Scraping
|
→ |
|
A Python library for automating interaction with websites.
|
|
|
|
| 9 |
urllib3
HTTP & Scraping
|
4,026 |
|
HTTP Clients
HTTP & Scraping
|
→ |
|
A HTTP library with thread-safe connection pooling, file post support, sanity friendly.
|
|
|
|
| 10 |
sumy
Content Extraction
|
3,686 |
|
Content Extraction
Web Scraping
HTTP & Scraping
|
→ |
|
A module for automatic summarization of text documents and HTML pages.
|
|
|
|
| 11 |
modoboa
HTTP & Scraping
|
3,488 |
|
Email
HTTP & Scraping
|
→ |
|
A mail hosting and management platform including a modern Web UI.
|
|
|
|
| 12 |
furl
HTTP & Scraping
|
2,801 |
|
HTTP Clients
HTTP & Scraping
|
→ |
|
A small Python library that makes parsing and manipulating URLs easy.
|
|
|
|
| 13 |
yagmail
HTTP & Scraping
|
2,725 |
|
Email
HTTP & Scraping
|
→ |
|
Yet another Gmail/SMTP client.
|
|
|
|
| 14 |
feedparser
Content Extraction
|
2,368 |
|
Content Extraction
Web Scraping
HTTP & Scraping
|
→ |
|
Universal feed parser.
|
|
|
|
| 15 |
html2text
Content Extraction
|
2,149 |
|
Content Extraction
Web Scraping
HTTP & Scraping
|
→ |
|
Convert HTML to Markdown-formatted text.
|
|
|
|
| 16 |
micawber
Content Extraction
|
674 |
|
Content Extraction
Web Scraping
HTTP & Scraping
|
→ |
|
A small library for extracting rich content from URLs.
|
|
|
|
| 17 |
httptap
HTTP & Scraping
|
497 |
|
HTTP Clients
HTTP & Scraping
|
→ |
|
Dissects an HTTP request into DNS, TCP, TLS, wait, and transfer phases and renders the timings as a waterfall.
|
|
|
|