Duplicate detection is a common problem in customer data management, as it is important to be able to identify and remove duplicate records to maintain the data’s integrity and reliability. Traditional approaches to duplicate detection rely on comparing individual fields within records, such as name, address, and phone number, to determine whether two records refer to the same customer. Additionally, it has been demonstrated that complex data makes it harder to identify duplicate records for businesses, which can increase processing time and leads to raising overhead cost. This research focuses on investigating the application of graph algorithms for identifying duplicate entries in customer data. Then, the scalability and algorithm performance of the graph database systems is evaluated in identifying duplicate customer data, contrasting their efficacy with that of relational databases. The findings revealed that PostgreSQL exhibits exceptional scalability and efficiency in handling large datasets, surpassing both Neo4J and MySQL. However, Neo4J excels in exact duplicate and near-duplicate detection algorithms, highlighting its proficiency in managing complex, interconnected data structures. In conclusion, the choice between PostgreSQL and Neo4J should be made based on specific task requirements, with PostgreSQL preferred for fast identification of similar items in large datasets, while Neo4J proves more suitable for tasks involving the discovery of communities within intricate networks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Scalability and Performance in Duplicate Detection: Relational Versus Graph Database

  • Noryanti Muhammad,
  • Muhammad Farhad Khaharruddin,
  • Mohd Izhar Firdaus

摘要

Duplicate detection is a common problem in customer data management, as it is important to be able to identify and remove duplicate records to maintain the data’s integrity and reliability. Traditional approaches to duplicate detection rely on comparing individual fields within records, such as name, address, and phone number, to determine whether two records refer to the same customer. Additionally, it has been demonstrated that complex data makes it harder to identify duplicate records for businesses, which can increase processing time and leads to raising overhead cost. This research focuses on investigating the application of graph algorithms for identifying duplicate entries in customer data. Then, the scalability and algorithm performance of the graph database systems is evaluated in identifying duplicate customer data, contrasting their efficacy with that of relational databases. The findings revealed that PostgreSQL exhibits exceptional scalability and efficiency in handling large datasets, surpassing both Neo4J and MySQL. However, Neo4J excels in exact duplicate and near-duplicate detection algorithms, highlighting its proficiency in managing complex, interconnected data structures. In conclusion, the choice between PostgreSQL and Neo4J should be made based on specific task requirements, with PostgreSQL preferred for fast identification of similar items in large datasets, while Neo4J proves more suitable for tasks involving the discovery of communities within intricate networks.