M3Bench: A Benchmark for Multi-model Databases on Analytical Query and In-Database Machine Learning Workloads

In the big data era, database management systems (DBMSs) are required to handle multiple data models (e.g., relational, document, graph, and key-value data) at the same time. Moreover, there is also an increasing demand for DBMSs to support the in-database execution of machine learning (ML) algorithms, commonly referred to as in-database ML. To meet these needs, many multi-model databases (MMDBs) that can handle different data models using a single storage engine, along with in-database ML integration, have been developed. However, it still lacks a unified benchmark to evaluate their performance on analytical query and in-database ML workloads. In this paper, we introduce M3Bench, a benchmark that fully tests the ability of DBMSs to perform analytical query and in-database ML tasks on multi-model data. M3Bench is built based on real-world data from an online education application. It consists of a flexible multi-model data generator, which can create relational, document, graph, and key-value data based on a scale factor and custom data distributions, and a suite of tasks that are representative analytical query and ML problems involving more than one data model. To validate the effectiveness of M3Bench, we benchmark four MMDBs, namely OrientDB, ArangoDB, Apache AGE, and AgensGraph, with the M3Bench workloads and highlight their advantages and disadvantages for each data model and task type.