CDMiA: Revealing Impacts of Data Migrations on Schemas in Multi-model Systems

To produce quality results from artificial intelligence (AI) pipelines, data consistency and structure must be finely controlled. This control must take into consideration the heterogeneity of data, and more specifically the heterogeneity of their models. Indeed, multi-model data are at the core of current systems, such as data lakes or polystores, that supply data for AI pipelines. To benefit the most of multi-model systems, data migrations are used to ingest data with a more suitable model than the original model, to apply operators available only for a specific model or to optimize queries, by sending data into a system capable of processing data faster than the original system, even when the migration time is included. However, data migrations are complex in a multi-model environment, in particular because the models do not support the same types of constraints. Thus, such constraints can be preserved, weakened, created or dropped during the migration process. Furthermore, the impact of a migration on a schema is contextual. Indeed, if a schema does not apply a constraint of its source model, it is not important that it is not supported in the destination model. Thus, it is essential to control data migrations, and to assess the impact of a migration on a schema. To do so, we propose the Categorical Data Migration Assessor tool (CDMiA), that relies on category theory to automatically detect the required creations and losses of constraints when migrating a schema from a model to another.