SemPart: Yet another RDF data partitioning strategy
摘要
RDF serves as a foundational standard for representing interconnected knowledge graphs. This has led to the emergence of large-scale knowledge graphs that must be efficiently stored and exploited to enable effective knowledge analytics and extract meaningful value. To address these requirements, several distributed RDF stores have been developed by both academia and industry. A critical challenge in such systems lies in efficient data partitioning, which directly affects query performance and system scalability. Our concise analysis of the most prominent distributed RDF stores that heavily rely on partitioning highlights three key dimensions: (a) query independence vs. dependence – including both workload-agnostic and workload-driven partitioning approaches, (b) partitioning granularity – at the level of triples, nodes, or larger fragments, and (c) resource usage – ranging from medium-memory to large-memory environments. This analysis reveals two major limitations in existing systems: Over-reliance on triple-based partitioning – many stores treat RDF data as isolated triples, disregarding the graph’s inherent structure. Notably, the most efficient systems are typically those deployed in large-memory settings. In contrast, disk-oriented distributed RDF stores, which make up the majority of existing systems, rely heavily on data partitioning to ensure efficiency. Current approaches tend to automate partitioning decisions rigidly, excluding domain experts from the process. This contrasts with relational database systems, where database administrators often fine-tune partitions for performance optimization. To bridge these gaps, we propose SemPart, a novel framework that integrates semantic-driven partitioning with administrator expertise. SemPart empowers administrators to define initial logical partitions using characteristic sets, which are then iteratively refined through various physical transformations to achieve an optimal balance between performance and scalability. This is facilitated by RDPAL (RDF Data Partitioning and Allocation Language), a comprehensive language offering primitives to manage and manipulate data partitions. We implement SemPart within PQDAG, a parallel extension of our centralized RDF store named QDAG. Experimental results show that PQDAG significantly outperforms state-of-the-art disk-based systems in query performance, thanks to the SemPart framework, while maintaining competitive preprocessing times. Additionally, it demonstrates comparable performance compared to leading in-memory triplestores.