Data Validation using Great Expectations

So far we looked at the need for and idea behind data validation, principles of data validation, and what happens when data isn’t validated. We also looked at key Python data validation libraries Pydantic and Pandera. Pydantic is one of the most utilized libraries in Python. We looked at creating a model class and instantiating an object from the class. We also looked at Pandera, a specialized open source Python library, with advanced data validation options like checking conditions at column levels and so on. In this chapter, we will be looking at Great Expectations, an entire data validation framework that is designed for managing data validation and testing for several production pipelines.