Scholarlink

<p>Effective task scheduling in Hadoop is a challenge because of heterogeneous cluster environments, unpredictable workload variations, and the absence of integrated methods that integrate precise workload prediction with adaptive, multi-objective optimisation. The existing schedulers are usually aimed at a single performance objective, such as execution time, energy efficiency, or fault tolerance, and they do not offer a unified model that can address all these objectives simultaneously. To address this gap, this study proposes a new approach to enhancing task management in Hadoop systems through the integration of deep learning ensembles and reinforcement learning. The framework uses a combination of SqueezeNet, MobileNetV2, and attention-based RNN models to predict computational load on the framework to process MapReduce job requests. It then uses load-balancing methods and hybrid optimisation algorithms to partition and group tasks, optimising resource utilisation, waiting time of jobs, and system throughput with multi-objective scheduling. Agents based on reinforcement learning, which is applied using deep Q-networks, dynamically rebalance loads in real time based on changes in workload. The experimental evaluation of a heterogeneous Hadoop cluster proves that the proposed approach is more efficient, scalable, and adaptable than existing approaches on various performance measures. This combined approach advances both predictive workload estimation and adaptive multi-objective optimisation in a single framework, making it a robust solution for modern big data environments.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Enhanced task management in Hadoop: integrating ensembled deep learning approach and reinforcement learning

  • 期刊论文

  • 发表日期：

    2025年11月3日

查看全文
  • Reshu Grover,
  • Manisha Agarwal

摘要

Effective task scheduling in Hadoop is a challenge because of heterogeneous cluster environments, unpredictable workload variations, and the absence of integrated methods that integrate precise workload prediction with adaptive, multi-objective optimisation. The existing schedulers are usually aimed at a single performance objective, such as execution time, energy efficiency, or fault tolerance, and they do not offer a unified model that can address all these objectives simultaneously. To address this gap, this study proposes a new approach to enhancing task management in Hadoop systems through the integration of deep learning ensembles and reinforcement learning. The framework uses a combination of SqueezeNet, MobileNetV2, and attention-based RNN models to predict computational load on the framework to process MapReduce job requests. It then uses load-balancing methods and hybrid optimisation algorithms to partition and group tasks, optimising resource utilisation, waiting time of jobs, and system throughput with multi-objective scheduling. Agents based on reinforcement learning, which is applied using deep Q-networks, dynamically rebalance loads in real time based on changes in workload. The experimental evaluation of a heterogeneous Hadoop cluster proves that the proposed approach is more efficient, scalable, and adaptable than existing approaches on various performance measures. This combined approach advances both predictive workload estimation and adaptive multi-objective optimisation in a single framework, making it a robust solution for modern big data environments.