AdpDM: Adaptive Data Model for Efficient Dynamic Management of Large-Scale High-Cardinality Time-Series Databases

With advancements in Internet of Things (IoT) and performance monitoring, time-series database have garnered significant attention. Time-series consists of two components: series keys and data points. Prior efforts focus on reducing the storage cost of data points. However, as the scale of data grows rapidly, storing series keys and their indexes consumes at most 75% storage costs, still leading to unacceptably high query latency. Furthermore, the growing cardinality of time-series also slows down the construction of real-time indexes. Previous methods often rely on a fixed data model or user-selected hot data mechanism for indexing, resulting in a small number of accessed indexes and high query latency. To pursue low storage costs, rapid real-time index construction and efficient query processing simultaneously, we propose a novel Ada ptive Data Model (AdpDM) for large-scale high-cardinality time-series data management. It involves three core modules: 1) a grouping algorithm based on the similarity of series keys is designed to group time-series; 2) a group-level inverted index instead of the traditional series-level inverted index is constructed to reduce storage costs and query latency; 3) for real-time group-level index construction, a grouping strategy reuse mechanism is proposed to improve ingestion throughput. We conduct extensive experiments to compare AdpDM with other advanced time-series databases, the results present that AdpDM reduces \(80\%\) storage cost of indexes, and \(90\%\) query latency while achieving \(2.8\times \) higher writing throughput.