HTTP & Scraping

Search every project in one place

Press / to search. Tap a tag to filter. Click any row for details.

Search and filter

Filtering for

Results

Row number Tags
Make websites accessible for AI agents with easy browser automation.
browser-use/github.com/browser-use/browser-use /
An open-source, LLM-friendly web crawler that provides lightning-fast, structured data extraction specifically designed for AI agents.
unclecode/github.com/unclecode/crawl4ai /
A fast high-level screen scraping and web crawling framework.
scrapy/github.com/scrapy/scrapy /
HTTP Requests for Humans.
psf/github.com/psf/requests /
Asynchronous HTTP client/server framework for asyncio and Python.
aio-libs/github.com/aio-libs/aiohttp /
A next generation HTTP client for Python.
encode/github.com/encode/httpx /
A tool for gathering text and metadata from the web, with built-in content filtering.
adbar/github.com/adbar/trafilatura /
A Python library for automating interaction with websites.
MechanicalSoup/github.com/MechanicalSoup/MechanicalSoup /
A HTTP library with thread-safe connection pooling, file post support, sanity friendly.
urllib3/github.com/urllib3/urllib3 /
A module for automatic summarization of text documents and HTML pages.
miso-belica/github.com/miso-belica/sumy /
A mail hosting and management platform including a modern Web UI.
modoboa/github.com/modoboa/modoboa /
A small Python library that makes parsing and manipulating URLs easy.
gruns/github.com/gruns/furl /
Yet another Gmail/SMTP client.
kootenpv/github.com/kootenpv/yagmail /
Universal feed parser.
kurtmckee/github.com/kurtmckee/feedparser /
Convert HTML to Markdown-formatted text.
Alir3z4/github.com/Alir3z4/html2text /
A small library for extracting rich content from URLs.
coleifer/github.com/coleifer/micawber /
Dissects an HTTP request into DNS, TCP, TLS, wait, and transfer phases and renders the timings as a waterfall.
ozeranskii/github.com/ozeranskii/httptap /

Contribute

Know a project that belongs here?

Tell us what it does and why it stands out.

Submit a project Star the repository