PMCKV: pipeline-based multi-compactions KV stores to improve the system performance

The log-structured merge tree (LSM-tree) in modern key-value (KV) storage systems needs compactions to move KV pairs from higher level to lower level to keep fully sorted ordering in each level and reclaim invalid KV pairs. Unfortunately, the system performance will be hurt because read–write amplification is caused by each compaction. In this paper, we have observed that read–write amplification exists between consecutive compactions within the same level of the LSM-tree. Motivated by this, we propose the compaction in advance strategy which triggers multiple consecutive compactions ahead of time when a certain level of the LSM-tree reaches its size limit. To improve the write performance, we design a novel pipeline parallel method based on the sorted string table (SST) file granularity. Additionally, we apply the compaction in advance strategy and traditional compaction, with the pipeline method, for low levels and high levels of the LSM-tree, respectively. Finally, we implement a prototype system called PMCKV based on LevelDB. The experimental results show that PMCKV, without increasing storage consumption, achieves a throughput approximately 1.2 \(-\) $-$ 1.7 times higher than LevelDB, 1.1 \(-\) $-$ 1.5 times higher than RocksDB, and 1.1 \(-\) $-$ 1.8 times higher than DiffKV, for write-intensive workloads. Moreover, PMCKV performs similarly to LevelDB and RocksDB for read-intensive workloads.