Data Analysis

pandas aims to be the fundamental high-level building block for practical data analysis in Python. Its DataFrame fits time series and tables with mixed column types, like an SQL table or a spreadsheet. It's built on NumPy to work with the rest of the scientific Python stack. In production code, the docs recommend .loc and .iloc over [] to select data. Set values in one .loc call, not chained indexing like df["foo"][mask] = 100. pandas keeps everything in memory, so a dataset that takes a sizable share of your RAM gets unwieldy, and the docs point you to other libraries.

Polars is built for multithreaded computing on a single machine, and its stricter API leads to fewer schema bugs. It has no index: rows are known by their position, so no index state can change what a query means. Make the lazy API your default: start from a function like scan_csv() or call .lazy(), then .collect() the result. That lets the optimizer filter rows and pick columns while reading the data. Stay eager for exploratory work, when you don't know yet what your query will look like. Write expressions inside select, with_columns, filter, and group_by, since Polars code that looks like pandas code likely runs slower than it should. For data that doesn't fit in memory, run the query on the streaming engine, which works through it in batches.

Ibis compiles one Python dataframe API into each backend's native language, mostly SQL, so the database or engine does the work. pandas' ecosystem page lists it for bridging local Python and remote databases. Start on the default DuckDB backend, as the tutorial recommends, then change the connection string to run the same code on PySpark, BigQuery, or Trino. Expressions are lazy: nothing runs until you call a method like to_pandas(), and only then does Ibis send the compiled query to the backend.

You can mix all three. Polars converts a DataFrame with to_pandas(), Ibis returns results as pandas or Polars DataFrames, and pandas can use PyArrow to trade data with Arrow-based libraries like Polars. Do the heavy work in Polars or Ibis, and hand a pandas DataFrame to the libraries that expect one.