Record linkage plays a crucial role in data integration by identifying and merging records that correspond to the same entity across different datasets. Traditional linkage methods are either lexically driven, relying on surface-level textual matching, or semantically driven, capturing contextual meaning. In this paper, we propose LSBlock, a hybrid blocking system that integrates lexical and semantic similarity search to enhance record linkage accuracy by improving recall while maintaining high precision. LSBlock employs minhash-based lexical similarity for efficient blocking key formulation and uses dense embeddings with cosine similarity to refine candidate matches. A Hierarchical Navigable Small-World (HNSW) index is leveraged for efficient approximate nearest neighbor searches. The retrieved results are further refined using two thresholds: one in the Jaccard space and another in the cosine similarity space, both derived after training a model using labeled matches. Our experimental evaluation on six real-world datasets demonstrates that LSBlock significantly outperforms state-of-the-art blocking techniques in terms of recall, precision, and F1-score, achieving a balanced trade-off between efficiency and effectiveness. LSBlock consistently achieves recall above 0.95 and precision above 0.65, when using structured data, outperforming its competitors by a large margin. These findings highlight the advantages of integrating lexical and semantic similarity, making LSBlock a robust solution for scalable record linkage.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

LSBlock: A Hybrid Blocking System Combining Lexical and Semantic Similarity Search for Record Linkage

  • Dimitrios Karapiperis,
  • Christos Tjortjis,
  • Vassilios Verykios

摘要

Record linkage plays a crucial role in data integration by identifying and merging records that correspond to the same entity across different datasets. Traditional linkage methods are either lexically driven, relying on surface-level textual matching, or semantically driven, capturing contextual meaning. In this paper, we propose LSBlock, a hybrid blocking system that integrates lexical and semantic similarity search to enhance record linkage accuracy by improving recall while maintaining high precision. LSBlock employs minhash-based lexical similarity for efficient blocking key formulation and uses dense embeddings with cosine similarity to refine candidate matches. A Hierarchical Navigable Small-World (HNSW) index is leveraged for efficient approximate nearest neighbor searches. The retrieved results are further refined using two thresholds: one in the Jaccard space and another in the cosine similarity space, both derived after training a model using labeled matches. Our experimental evaluation on six real-world datasets demonstrates that LSBlock significantly outperforms state-of-the-art blocking techniques in terms of recall, precision, and F1-score, achieving a balanced trade-off between efficiency and effectiveness. LSBlock consistently achieves recall above 0.95 and precision above 0.65, when using structured data, outperforming its competitors by a large margin. These findings highlight the advantages of integrating lexical and semantic similarity, making LSBlock a robust solution for scalable record linkage.