Data Ingestion / ETL

dlt loads data from messy sources into well-structured datasets: it infers the schema and data types, normalizes the data, and handles nested structures. Start a project with dlt init rest_api duckdb, which sets up a pipeline script with a REST API source and a DuckDB destination. Build and test on DuckDB, then switch out the destination when you deploy. Beyond the built-in sources, any Python iterable can feed it, since a resource is just a generator. dlt runs anywhere Python runs, so schedule it with the orchestrator you already have.

AWS SDK for pandas extends pandas to AWS: import it as awswrangler, and it connects DataFrames to S3, Athena, Glue, Redshift, and DynamoDB. Its quick start stores a DataFrame on S3 as a Parquet dataset with wr.s3.to_parquet(), passing a database and table name. Then wr.athena.read_sql_query() queries that table with SQL.

yfinance offers a Pythonic way to fetch financial and market data from Yahoo Finance. Its quick start uses Ticker for one symbol's prices, financial statements, and options, and download for prices across several symbols at once.

AKShare aims to simplify fetching financial data, with one function call per dataset, like ak.stock_zh_a_hist() for the daily history of a Chinese stock. Its README credits mostly Chinese exchanges and finance sites as its sources. Its docs are in Chinese, and each data interface comes with an example you can copy and paste.

EdgarTools turns any SEC filing into a typed Python object, so a 10-K's revenue is one line instead of an afternoon of XBRL parsing. EDGAR requires an email with every request, so set your identity with set_identity() before anything else. Everything starts with a Company or a Filing. Company("AAPL").get_financials().income_statement() returns a standardized income statement, and .obj() turns a filing into an object built for its form, with its data as pandas DataFrames.

OpenBB is a "connect once, consume everywhere" layer over financial data sources. Each provider package connects one source and adds its own namespace to the obb client. The same commands run from Python, as a REST API, and as MCP tools. A call returns an OBBject, and to_dataframe() turns it into a pandas DataFrame.

Before you build on financial data, check whose terms you're under. yfinance isn't affiliated with Yahoo and is meant for research and education, and Yahoo's API is for personal use only. AKShare's data is for academic research only. OpenBB hosts no data itself: each provider sets its own coverage and terms of use.