HTTP & Scraping

Listed by section, in editorial order. Click a column header to re-sort the whole list.

Press / to search. Tags open their category. Click a row for details.

Search and filter

Results

Row number Tags

HTTP Clients 6 projects

Requests makes synchronous HTTP calls simple. A Python HTTP client for async code or HTTP/2 means httpx2, whose API stays requests-compatible.

HTTP Requests for Humans.
psf/github.com/psf/requests /54,578 stars /
Asynchronous HTTP client/server framework for asyncio and Python.
aio-libs/github.com/aio-libs/aiohttp /16,597 stars /
HTTP/1.1 and HTTP/2 client with sync and async APIs, maintained by Pydantic (httpx fork).
pydantic/github.com/pydantic/httpx2 /1,608 stars /
An HTTP library with thread-safe connection pooling, file post, and more.
urllib3/github.com/urllib3/urllib3 /4,081 stars /
A next-generation HTTP client for Python.
encode/github.com/encode/httpx /15,712 stars /
Yet another URL library.
aio-libs/github.com/aio-libs/yarl /1,503 stars /

Web Scraping 8 projects

Most crawls you build with a Python web scraping library fit Scrapy: you write selectors, it schedules requests. Crawl4AI hands pages to an LLM as Markdown.

Make websites accessible for AI agents with easy browser automation.
browser-use/github.com/browser-use/browser-use /119,599 stars /
A fast high-level web crawling and scraping framework.
scrapy/github.com/scrapy/scrapy /65,122 stars /
An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents.
unclecode/github.com/unclecode/crawl4ai /86,615 stars /
A fast and token-efficient browser automation SDK to extract data and perform self-healing actions on web pages.
browserbase/github.com/browserbase/stagehand /25,882 stars /
An adaptive scraping framework whose selectors relocate elements after page changes, with stealth fetchers and spiders.
D4Vinci/github.com/D4Vinci/Scrapling /90,480 stars /
Convert HTML to Markdown, with customizable tag handling.
Universal feed parser.
kurtmckee/github.com/kurtmckee/feedparser /2,453 stars /
A tool for gathering text and metadata from the web, with built-in content filtering.
adbar/github.com/adbar/trafilatura /7,046 stars /

Email 3 projects

Sending from Django, you learn no new Python email library API: django-anymail runs send_mail(). Async code sends mail through aiosmtplib.

An asyncio SMTP client.
cole/github.com/cole/aiosmtplib /436 stars /
Django email backends and webhooks for transactional email services such as Amazon SES, Brevo, Mailgun, Postmark, and Resend.
anymail/github.com/anymail/django-anymail /1,915 stars /
Yet another Gmail/SMTP client.
kootenpv/github.com/kootenpv/yagmail /2,739 stars /

Know a project that belongs here?

Tell us what it does and why it stands out.