Data Validation

Pydantic takes its schema from Python type hints, so your type checker and IDE read the same code it validates. Subclass BaseModel and annotate its fields, then pass untrusted data to the model: if no ValidationError is raised, every field matches its declared type. Your models can also generate JSON Schema for self-documenting APIs and other tools.

Pandera validates pandas, Polars, PySpark, and other dataframes against one schema. Write that schema as a DataFrameModel, much the way you'd define a Pydantic model, with a typed field per column. Then decorate your pipeline functions with check_types(), which validates both inputs and outputs from their type annotations.

jsonschema implements the JSON Schema specification, so the schema is a JSON document rather than Python code. That suits a schema that has to work outside Python too, since JSON Schema establishes a common language for data exchange across systems. Declare $schema in each schema to identify which version it's written for.

Great Expectations is a framework for describing data with expressive tests and validating data against them, from databases to files in cloud storage. Every script starts by creating a Data Context. Group your production Expectations into Expectation Suites, and run those through a Checkpoint, which can send email or Slack alerts and write the results to Data Docs, a human-readable report.

Records and tables need different validators, but the picks work together. A Pandera DataFrameModel works as a field on a Pydantic model, with Pydantic checking the built-in types next to it. Pandera's maintainer suggests Pandera for in-memory dataframes and Great Expectations for data on disk. Pandera needs zero configuration, while Great Expectations takes some setup up front.