Data missingness and inconsistency are common issues in large databases, especially those collected over short periods, manually entered and lacking quality control tools during and after data collection and entry. In Vietnam, the National Resident Database has collected 104 million records within two years, currently being utilized to support economic and social development applications. This study aims to develop a tool based on big data platforms to first detect issues of missing and inconsistent data and then suggest corrective solutions. Based on comparing information among individuals with spousal and parental relationships, the study successfully develops a framework that includes a rich set of rules for detecting and suggesting corrections for missing and inconsistent data. Additionally, to assist staff in data updates, a family tree visualization tool has been developed. Experimental results indicate that the developed framework operates efficiently, works on large databases (data of over 100 million citizens) and successfully identifies and suggests solutions for missing and inconsistent data issues.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Framework for Detecting and Suggesting Corrections for Missing and Inconsistent Data in Vietnam’s National Residents Database

  • Tuan-Anh Nguyen,
  • Quang-Dung Pham,
  • Quoc-Trung Bui

摘要

Data missingness and inconsistency are common issues in large databases, especially those collected over short periods, manually entered and lacking quality control tools during and after data collection and entry. In Vietnam, the National Resident Database has collected 104 million records within two years, currently being utilized to support economic and social development applications. This study aims to develop a tool based on big data platforms to first detect issues of missing and inconsistent data and then suggest corrective solutions. Based on comparing information among individuals with spousal and parental relationships, the study successfully develops a framework that includes a rich set of rules for detecting and suggesting corrections for missing and inconsistent data. Additionally, to assist staff in data updates, a family tree visualization tool has been developed. Experimental results indicate that the developed framework operates efficiently, works on large databases (data of over 100 million citizens) and successfully identifies and suggests solutions for missing and inconsistent data issues.