ESGTabQA: Open-Domain Question Answering on High-Cardinality ESG Tables

The analysis of Environmental, Social, and Governance (ESG) data presents significant challenges for traditional question answering (QA) systems. These systems, typically designed for small and well-structured tables, struggle to handle the high-cardinality, ambiguity, and constant updates that characterize real-world ESG datasets. To address this, we present ESGTabQA, a table-based question-answer system capable of handling online, open-domain, natural language queries over high-cardinality ESG tables. The system integrates three key components: an MPNet-based retriever to identify relevant data segments, the TAPAS model to extract answers from tables, and GPT-4 to generate clear responses in natural language. A central innovation of our system is an entity-specific query decomposition that breaks down complex multi-entity queries into parallel, independent sub-queries that align with the data’s structure. This is particularly effective for comparative questions, allowing the system to locate and reason over data spread across multiple sections of a large dataset. We evaluate ESGTabQA on 32 real-world ESG queries, covering simple, ambiguous, and comparative scenarios. The proposed system achieves strong results, including a semantic similarity score of 0.876, 79.5% keyword inclusion, and an F1-score of 0.597. These results demonstrate ESGTabQA’s ability to scale QA to real-world datasets, supporting more efficient information retrieval and analysis in evolving data environments.