In enterprise settings, building effective machine learning (ML) products relies on solid data engineering, data quality, and data security. This work focuses on data pipeline engineering along with quality control and security in building a solid ML product. It is enterprise data engineering that transforms unstructured enterprise data into usable, consistent, scalable, and performant structured data. Data quality provides consistency, accuracy, completeness, and timeliness of data which is necessary for the ML predictive models and analytical outcomes. Moreover, enterprise scale ML applications need a security design that provides control of access and data, encryption, anonymization, and compliance with data policies such as GDPR and HIPAA. The study addresses common enterprise concerns including diverse data repositories, disparate schemas, NULLs, and regulatory gaps that diminish the effectiveness of ML products. The study then discusses the available methodologies and best practices such as ETL (Extract, Transform, Load) pipelines, schema validation, data lineage tracking, and secure machine learning for data engineering. An architecture is proposed that integrates the data engineering with security and data quality processes. The study has reported its results on a synthetic enterprise data set that has evidenced improvements in accuracy, robustness, and compliance of the models. The current paper demonstrates the synergistic roles of data engineering tested with quality control and secured as per the current best practices in the industry in ML product design with a preview of the tabular results and the architectural drawings complemented with visualization of the performances attained. In the form of hand-on guidelines and tangible outcomes, bolstered with a precise path, the current paper constitutes a first of its kind contribution to the field by giving practitioners the necessary tools and framework to employ machine learning in industry with security, explainability, and compliance in its design.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Data Engineering, Data Quality, and Security for ML Product Development in Enterprise Environments

  • Chirag Agrawal,
  • Udaya Veeramreddygari,
  • Supraja Chinthala

摘要

In enterprise settings, building effective machine learning (ML) products relies on solid data engineering, data quality, and data security. This work focuses on data pipeline engineering along with quality control and security in building a solid ML product. It is enterprise data engineering that transforms unstructured enterprise data into usable, consistent, scalable, and performant structured data. Data quality provides consistency, accuracy, completeness, and timeliness of data which is necessary for the ML predictive models and analytical outcomes. Moreover, enterprise scale ML applications need a security design that provides control of access and data, encryption, anonymization, and compliance with data policies such as GDPR and HIPAA. The study addresses common enterprise concerns including diverse data repositories, disparate schemas, NULLs, and regulatory gaps that diminish the effectiveness of ML products. The study then discusses the available methodologies and best practices such as ETL (Extract, Transform, Load) pipelines, schema validation, data lineage tracking, and secure machine learning for data engineering. An architecture is proposed that integrates the data engineering with security and data quality processes. The study has reported its results on a synthetic enterprise data set that has evidenced improvements in accuracy, robustness, and compliance of the models. The current paper demonstrates the synergistic roles of data engineering tested with quality control and secured as per the current best practices in the industry in ML product design with a preview of the tabular results and the architectural drawings complemented with visualization of the performances attained. In the form of hand-on guidelines and tangible outcomes, bolstered with a precise path, the current paper constitutes a first of its kind contribution to the field by giving practitioners the necessary tools and framework to employ machine learning in industry with security, explainability, and compliance in its design.