Controllable Markov Chains

In this chapter, we describe a class of discrete-time, controllable, and ergodic Markov chains. In these concepts, time and space are discrete. We start by outlining the fundamental model. Its single-step transition probabilities is then utilized to determine if a Markov chain is ergodic. In order to convert the nonlinear optimization Markov problem into a linear one and make the problem tractable, we propose to use an auxiliary c-variable. With such a setup, a discrete time Markov chain simulation is shown. A piece of the illustrative Markov chain laws for discrete time is constructed in the end.