Purpose <p>The Einstein Probe (EP) satellite is equipped with the Wide-field X-ray Telescope (WXT) and the Follow-up X-ray Telescope (FXT). The EP-FXT Workflow Management System is part of the FXT-SDC and is responsible for managing the workflow involved in data product generation. It is utilized for the fully automated processing of FXT data within the pipeline for generating data products.</p> Methods <p>The EP-FXT Workflow Management System invokes computing servers by leveraging the computing resource scheduling platform HTCondor, constructs workflows through the workflow orchestration platform Dolphineschedule, detects data via the central control website, automatically triggers workflows, monitors the workflow processing procedure, and provides a feedback and redo mechanism.</p> Results <p>The workflow processing was automatically initiated 16,900 times for 4000 observations within 270 days, and approximately 40,000 files are archived daily. The system handles both standalone servers and computing clusters. Under the circumstances of sufficient resources where tasks do not need to wait, when utilizing the resources of standalone server, the average time for task switching is 1.4&#xa0;s. In contrast, when using the computing cluster, the average time for task switching is 2.7&#xa0;s. Regarding the Observation Data Workflow, 26.91% of the tasks are completed within 10&#xa0;min, and 71.7% are completed within 100&#xa0;min.</p> Conclusion <p>The EP-FXT Workflow Management System is capable of effectively expediting the generation of data products.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

EP-FXT workflow management system

  • Ge Ou,
  • Shu-mei Jia,
  • Hong-mei Zhang,
  • Li-ming Song,
  • Jian-chuan Gao,
  • Hai-Sheng Zhao,
  • Ju Guan,
  • Cheng-kui Li,
  • Juan Zhang,
  • Jin Wang

摘要

Purpose

The Einstein Probe (EP) satellite is equipped with the Wide-field X-ray Telescope (WXT) and the Follow-up X-ray Telescope (FXT). The EP-FXT Workflow Management System is part of the FXT-SDC and is responsible for managing the workflow involved in data product generation. It is utilized for the fully automated processing of FXT data within the pipeline for generating data products.

Methods

The EP-FXT Workflow Management System invokes computing servers by leveraging the computing resource scheduling platform HTCondor, constructs workflows through the workflow orchestration platform Dolphineschedule, detects data via the central control website, automatically triggers workflows, monitors the workflow processing procedure, and provides a feedback and redo mechanism.

Results

The workflow processing was automatically initiated 16,900 times for 4000 observations within 270 days, and approximately 40,000 files are archived daily. The system handles both standalone servers and computing clusters. Under the circumstances of sufficient resources where tasks do not need to wait, when utilizing the resources of standalone server, the average time for task switching is 1.4 s. In contrast, when using the computing cluster, the average time for task switching is 2.7 s. Regarding the Observation Data Workflow, 26.91% of the tasks are completed within 10 min, and 71.7% are completed within 100 min.

Conclusion

The EP-FXT Workflow Management System is capable of effectively expediting the generation of data products.