Introduction to Concurrency Programming and Dask

We have discussed earlier that processing large datasets effectively and efficiently is important. As datasets continue to grow over time, traditional single-thread processing can be seen as a limitation. Python, by default, has a global interpreter lock (GIL) that allows only one thread to hold the interpreter at a given point in time. While this design ensures the integrity of computations submitted, it would be much more effective to use many threads within a multi-core CPU that is commonly available today. In this chapter, you will learn concepts of parallel programming, concurrent programming, and Dask, a Python library that supports distributed processing and works around the global interpreter lock limitation by using multiple processes. Dask also supports various data processing libraries that we have seen in earlier chapters.