Scholarlink

In today’s data-driven companies, the value of large data lakes rests on the quality of high-grade, accurate metadata—though tagging is an exhaustive and error-prone undertaking. This article describes an AI-based metadata enrichment system that employs Transformer-based NLP models to generate descriptive tags and inter-entity relationships for raw data assets automatically. We demonstrate our approach on the Enron Email Dataset from a publicly available release, training a BERT model to identify central entities (e.g., originator, recipient, subject) and infer contextual meta-data (e.g., project code, sensitivity). Integrated with an emulated Azure Data Lake environment, our system live-annotates onboarding emails, achieving a 28% boost in metadata span and a 15% improvement in search relevance over heuristics base-lines. A user test with data stewards also shows 40% reduction in effort of manual curation and high satisfaction with tag accuracy. These results suggest the potential of Transformer-powered enrichment to make enterprise data lakes more discoverable, authoritative, and ready for downstream analytics.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

AI-Augmented Metadata Enrichment in Enterprise Data Lakes Using Transformer-Based NLP

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Naveen Kolli,
  • Bharath Reddy Thipi Reddy,
  • Saravanan Jayakumar,
  • Santhosh Kumar Veeramalla

摘要

In today’s data-driven companies, the value of large data lakes rests on the quality of high-grade, accurate metadata—though tagging is an exhaustive and error-prone undertaking. This article describes an AI-based metadata enrichment system that employs Transformer-based NLP models to generate descriptive tags and inter-entity relationships for raw data assets automatically. We demonstrate our approach on the Enron Email Dataset from a publicly available release, training a BERT model to identify central entities (e.g., originator, recipient, subject) and infer contextual meta-data (e.g., project code, sensitivity). Integrated with an emulated Azure Data Lake environment, our system live-annotates onboarding emails, achieving a 28% boost in metadata span and a 15% improvement in search relevance over heuristics base-lines. A user test with data stewards also shows 40% reduction in effort of manual curation and high satisfaction with tag accuracy. These results suggest the potential of Transformer-powered enrichment to make enterprise data lakes more discoverable, authoritative, and ready for downstream analytics.