Data Processing and Analysis

The last several chapters covered the main topics of traditional scientific computing. These topics provide a foundation for most computational work. Starting with this chapter, let’s move on to explore data processing and analysis, statistics, and statistical modeling. First, we look at the pandas data analysis library. This library provides convenient data structures for representing series and tables of data and makes it easy to transform, split, merge, and convert data. These are important steps in the process of cleansing raw data into a tidy form suitable for analysis. The Pandas library builds on top of NumPy. It complements it with features that are particularly useful when handling data, such as labeled indexing, hierarchical indices, data alignment for comparison and merging of datasets, handling of missing data, and much more. As such, the pandas library has become a de facto standard library for high-level data processing in Python, especially for statistics applications. The pandas library contains only limited support for statistical modeling (namely, linear regression). Other packages are available for more involved statistical analysis and modeling, such as statsmodels, patsy, and scikit-learn, which are covered in later chapters. However, for statistical modeling with these packages, pandas can still be used for data representation and preparation. The pandas library is, therefore, a key component in the software stack for data analysis with Python.