Discrete-Time Markov Chains—Part I
摘要
A Markov process is a stochastic process used to model the events, where the system moves between states and the probability of transitioning from one state to the next state depends solely on the current state, with no influence from the previous visited states. Markov processes are categorized based on the nature of their state and parameter space. When the state space is discrete, the process is called a Markov chain (MC). If both the state and parameter space are discrete, it is specifically known as a discrete-time Markov chain (DTMC). In this chapter, the definition and basic properties of DTMC are presented. Also, limiting distribution, stationary distribution and steady-state analysis of the DTMCs are also considered. Moreover, this chapter and Chap. 3 both are devoted to an elaboration of the basic concepts and methods of DTMC.