A core challenge data consumers face in large enterprise data lakes is to efficiently understand and reuse data assets due to the prevalence of technical, often vendor-specific, naming conventions. This paper introduces a novel Gen-AI-based co-pilot designed to automate and significantly improve the process of semantic mapping within enterprise data lakes. The proposed solution leverages an agentic workflow powered by Large Language Models (LLMs), a vector DB for similarity search, and a Knowledge Graph (KG) representing Bosch’s internal vocabularies and semantic data models. Empirical evaluation on real-world data demonstrates a sufficiently high accuracy in automatically suggesting correct semantic mappings, leading to significant time savings for data stewards, and enhanced data transparency and interoperability in line with FAIR data principles [2].

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Gen-AI Co-pilot for Semantic Mapping in Large Enterprise Data Lakes

  • Stefan Schmid,
  • Lavdim Halilaj,
  • Yi Tan

摘要

A core challenge data consumers face in large enterprise data lakes is to efficiently understand and reuse data assets due to the prevalence of technical, often vendor-specific, naming conventions. This paper introduces a novel Gen-AI-based co-pilot designed to automate and significantly improve the process of semantic mapping within enterprise data lakes. The proposed solution leverages an agentic workflow powered by Large Language Models (LLMs), a vector DB for similarity search, and a Knowledge Graph (KG) representing Bosch’s internal vocabularies and semantic data models. Empirical evaluation on real-world data demonstrates a sufficiently high accuracy in automatically suggesting correct semantic mappings, leading to significant time savings for data stewards, and enhanced data transparency and interoperability in line with FAIR data principles [2].