Entity Resolution (ER) is the process of locating records that represent the same real-world entity, within a single or across different datasets. Various techniques are used for the implementation of the ER process, like blocking, filtering, and matching, in order to improve its performance and effectiveness. However, ER faces new challenges in the age of big data analytics, since traditional methods of handling data have not proved very efficient. In this work, we study the ER process and focus on further improving the efficiency of blocking techniques by leveraging the capabilities of big data parallel processing platforms. Specifically, we introduce parallelized suffix array blocking with Bloom filters using Apache Spark and explore its performance compared to a serial implementation. Our evaluation results show significant improvement with the parallel approach being up to 5 times faster.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Parallel Suffix Array Blocking for Efficient Entity Resolution Based on Spark

  • Vasileios Tsogkas,
  • Georgia Koloniari

摘要

Entity Resolution (ER) is the process of locating records that represent the same real-world entity, within a single or across different datasets. Various techniques are used for the implementation of the ER process, like blocking, filtering, and matching, in order to improve its performance and effectiveness. However, ER faces new challenges in the age of big data analytics, since traditional methods of handling data have not proved very efficient. In this work, we study the ER process and focus on further improving the efficiency of blocking techniques by leveraging the capabilities of big data parallel processing platforms. Specifically, we introduce parallelized suffix array blocking with Bloom filters using Apache Spark and explore its performance compared to a serial implementation. Our evaluation results show significant improvement with the parallel approach being up to 5 times faster.