Scholarlink

<p>In data stream applications, a query is created using a series of operators connected by unidirectional pipelines. Since those operators may be deployed in a shared-nothing distributed environment, it is necessary to have a valid restore point in case an operator fails. This ensures consistency of the states among operators after recovering from failures. Typical fault-tolerant approaches periodically capture global snapshots, encompassing all operator states. In the recovery process, all operators must be reinstated to the most recent global snapshot. This global dependency affects the recovery performance and the required computational costs for recovery. We propose fault-tolerant schemes for stateful stream processing that localizes recovery to a subset of operators. Additionally, we introduce a partial snapshot mechanism to capture the states required for local recovery efficiently. This snapshot records only a subset of operator states instead of global states. Our implemented recovery schemes in Apache Flink show improved recovery times by up to more than 50 percent.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Local recovery and partial snapshot in distributed stateful stream processing

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年6月30日

查看全文
  • Takdir,
  • Hiroyuki Kitagawa,
  • Toshiyuki Amagasa

摘要

In data stream applications, a query is created using a series of operators connected by unidirectional pipelines. Since those operators may be deployed in a shared-nothing distributed environment, it is necessary to have a valid restore point in case an operator fails. This ensures consistency of the states among operators after recovering from failures. Typical fault-tolerant approaches periodically capture global snapshots, encompassing all operator states. In the recovery process, all operators must be reinstated to the most recent global snapshot. This global dependency affects the recovery performance and the required computational costs for recovery. We propose fault-tolerant schemes for stateful stream processing that localizes recovery to a subset of operators. Additionally, we introduce a partial snapshot mechanism to capture the states required for local recovery efficiently. This snapshot records only a subset of operator states instead of global states. Our implemented recovery schemes in Apache Flink show improved recovery times by up to more than 50 percent.