Discrete Markov Processes and Numerical Algorithms for Markov Chains

This chapter states the necessary classical results on discrete-time Markov processes and presents some approaches for determining the basic probabilistic characteristics of finite state space Markov chains. The main focus is on the elaboration of efficient numerical algorithms for computing the state-time probabilities, the limiting and differential matrices, as well as the average and expected total discounted rewards for discrete-time Markov processes. New algorithms for determining the limiting and differential matrices for such processes based on the z-transform are developed, and innovative procedures for calculating the state-time probabilities based on dynamic programming are substantiated.