Leveraging Azure Data Factory for COVID-19 Data Ingestion, Transformation, and Reporting

The COVID-19 pandemic has necessitated the collection and analysis of large volumes of data from various sources to understand its impact and make informed decisions. This research paper explores the utilization of Azure Data Factory, a managed cloud solution, for integrating and reporting COVID-19 data. The study focuses on two primary objectives. Firstly, it establishes a data platform using Azure Data Factory to enable data scientists to leverage machine learning models for predicting the spread of the virus and gaining insightful findings. This work demonstrates Azure Data Lake Gen2 as a reliable storage repository for ingesting and storing data. Furthermore, it explores the data transformation and analysis capabilities of Azure Data Factory using Data Flows, HDInsight, and Azure Databricks. The second objective revolves around the creation of a reporting platform for data analysts. This study illustrates how Azure Data Factory orchestrates the extraction, transformation, and loading of selected COVID-19 data into a SQL database. This subset of data is then leveraged to develop a Power BI report, enabling data analysts to visualize and report COVID-19 trends efficiently. Through a comprehensive analysis and practical implementation of Azure Data Factory, this research highlights its efficacy in handling COVID-19 data integration and reporting tasks. It sheds light on the data ingestion, transformation, and storage options provided by Azure Data Factory. It also showcases the seamless integration with other Azure services, including Data Flows, HDInsight, Azure Databricks, SQL databases, and Power BI.