Distributed Computing

joblib is a package for parallel computing and disk-based caching that leaves your code as unmodified as possible. You write a parallel for loop as a generator expression: Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10)). By default, joblib runs the calls in separate worker processes. When one machine isn't enough, switch its backend, and the same loop runs on a Dask, Ray, or Spark cluster. scikit-learn's n_jobs runs on joblib too, so it follows the backend you pick.

Dask scales pandas, scikit-learn, and NumPy workflows with minimal rewriting. A Dask DataFrame is a collection of pandas dataframes on different computers, and Dask Arrays parallelize NumPy. Dask runs without any setup on your laptop. Its LocalCluster follows the same interface as every other Dask cluster manager, so you swap it out when you're ready to scale up.

Ray is a unified framework for scaling AI and Python applications. Its libraries each distribute one ML task: Ray Data, Train, Tune, Serve, and RLlib. Ray Core runs your own code: decorate a function with @ray.remote, call it with .remote(), and fetch the result with ray.get(). For workers that keep state between calls, decorate a class the same way to get an actor. Ray runs on one machine with ray.init(), and on several nodes once you deploy a Ray cluster. Ray Data suits GPU workloads for deep learning inference better than Spark does, but unlike Spark, it has no SQL interface.

PySpark is the Python API for Apache Spark, for large-scale data processing. Dask's own comparison suggests Spark when you prefer SQL, run mostly JVM infrastructure, or want an all-in-one solution. PySpark's docs recommend DataFrames over RDDs, so Spark builds the most efficient query for you. Write it in SQL or the DataFrame API, whichever you think in, and switch between the two as you go. Installing PySpark with pip is for local use or as a client that connects to a cluster, not for setting up the cluster itself. The pyspark package also needs Java.

mpi4py provides Python bindings for MPI, the Message Passing Interface, so your Python code runs across workstations, clusters, and supercomputers. Lowercase methods like comm.send pass any picklable Python object, and uppercase ones like comm.Send pass NumPy arrays the fast way. Run your script with mpiexec -n 4 python -m mpi4py script.py, so an unhandled exception aborts the whole MPI run instead of deadlocking. mpi4py runs on an MPI implementation like MPICH or Open MPI, and in production its docs recommend a custom-built or system-provided one.

Start on one machine: parallelism brings extra complexity and overhead, and often you don't need it. When you do scale out, collect results at the end, since Dask's compute() and Ray's ray.get() both block until the work finishes. Call compute() once rather than in a loop, and ray.get() as late as possible.