A new framework is proposed to remediate the current real-time data quality by incorporating the anomaly signature learning, the traversal of causal dependency, resource-sensitive implementation, and fault resilient dynamic-data-pipe orchestration. As opposed to traditional systems which re-execute entire workflows when anomalies do occur (e.g. null spikes or schema drift), this method encodes context rich signatures and uses time-decayed similarity scoring to compare with a historical repository. Once a match is observed, the associated remediation path is reused, and otherwise, a minimal subgraph is chosen through casually-aware DAG search. Metadata about dependencies and resource costs is stored within each node of the DAG making it possible to execute even when there are limited compute budgets. There is a checkpoint such that partial rollbacks occur in case of downstream failures; previous successful states are maintained. There is also support in the architecture to support complex DAG topologies arising with branching and joins enabling fine grained remediation. The system was tested against Numenta Anomaly Benchmark, improving the execution savings by more than 60% and doubling the remediation rate, and performed regardless of external models and human intervention.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Signature-Based Anomaly-Aware DAG Execution for Efficient Real-Time Data Quality Remediation

  • Rakesh Keshava,
  • Arun Kumar Elengovan,
  • Nandagopal Seshagiri,
  • Lahari Putty

摘要

A new framework is proposed to remediate the current real-time data quality by incorporating the anomaly signature learning, the traversal of causal dependency, resource-sensitive implementation, and fault resilient dynamic-data-pipe orchestration. As opposed to traditional systems which re-execute entire workflows when anomalies do occur (e.g. null spikes or schema drift), this method encodes context rich signatures and uses time-decayed similarity scoring to compare with a historical repository. Once a match is observed, the associated remediation path is reused, and otherwise, a minimal subgraph is chosen through casually-aware DAG search. Metadata about dependencies and resource costs is stored within each node of the DAG making it possible to execute even when there are limited compute budgets. There is a checkpoint such that partial rollbacks occur in case of downstream failures; previous successful states are maintained. There is also support in the architecture to support complex DAG topologies arising with branching and joins enabling fine grained remediation. The system was tested against Numenta Anomaly Benchmark, improving the execution savings by more than 60% and doubling the remediation rate, and performed regardless of external models and human intervention.