The Databricks Data Intelligence Platform

In the previous chapters, we learned about the Databricks lakehouse, which essentially means storing all your data in open storage in an open format with Unity Catalog providing a single governance layer and Databricks providing features to enable all use cases such as data engineering, data science, streaming, and warehousing. With the advent and popularity of GenAI and LLMs since 2023, Databricks has integrated them into its platform. The Databricks data intelligence platform (see Figure 14-1) combines the lakehouse platform and AI/LLMs to add the “data intelligence” engine that understands the uniqueness of your data and uses that understanding across everything in the platform.