Enhancing Robotic Systems for Revolutionizing Healthcare Using Markov Decision Processes

This work is part of a research project carried out during the COVID-19 pandemic, involving the design and realization of an autonomous mobile hospital robot. Many real-world robotic tasks suffer from the critical characteristics: Noisy sensing, imperfect control, and environment changes. The Markov decision process MDP and its variants provide a mathematically based framework for modeling and solving robot decision and control tasks under uncertainty. This paper presents a review of Markov Decision Processes (MDPs) and their variants in the Control of Robotic systems. We begin by introducing the basic concepts of MDPs and their algorithms for solving completely observable decision problems, including value iteration and policy iteration. We then discuss the challenges associated with partially observable decision problems POMDPs. We review various approaches for solving POMDPs, including belief-state planning and Monte Carlo tree search. Finally, we discuss the concept of POMDP augmentation, which involves incorporating additional information into the decision-making process to improve performance. We present several examples of POMDP augmentation techniques, including the use of deep neural networks and transfer learning.