Getting Started with Data Validation using Pydantic and Pandera

Data validation has been in existence for a long time. During the early adoption of relational database management systems (RDBMSs), basic validation rules were defined within a limited scope. As SQL evolved within the relational systems, it provided more opportunities with respect to specifying better validation rules by writing SQL code. In the modern days of big data and machine learning, data validation has occupied greater relevance. The quality of data directly affects the insights and intelligence derived from analytical models that are built using them. In this chapter we will explore two major data validation libraries, namely, Pydantic and Pandera, and delve into features, capabilities, and practical applications.