Bridging language to visuals: towards natural language query-to-chart image retrieval

Given a natural language query, mining a relevant chart image, i.e., the one that contains the answer to the query, is an overlooked problem in the literature. Our study explores this novel problem. Consider an example of retrieving relevant chart images for a query: Which Indian city has the highest annual rainfall over the past decade?. Retrieving relevant chart images for such natural language queries necessitates a deep semantic understanding of chart images. Towards addressing this problem, in this work, we make two key contributions: (a) We present a dataset, namely WebCIRD (or Web Chart Image Retrieval) for studying this problem, and (b) propose a solution viz. ChartSemBERT that offers a deeper semantic understanding of chart images for effective natural language-to-chart image retrieval. Our proposed approach yields remarkable performance improvements compared to the existing baselines, achieving R@10 as 86.9%.