Q-Learning Based Methods for Dynamic Treatment Regimes

Precision medicine seeks to find the optimal treatments tailored to individual patient characteristics. Dynamic treatment regimes are sequences that formalize the process of decision-making by mapping from patients’ observable information to a recommended treatment. Q-learning is a popular approach for estimating an optimal treatment regime. It is closely related to the regression-based analysis in statistics and belongs to the family of reinforcement learning methods. In this chapter, we provide an introduction of Q-learning based methods for the estimation of dynamic treatment regimes. We start with the formal precision medicine framework, followed by the introduction of reinforcement learning. We then delve into Q-learning based methods for dynamic treatment regime in the finite time horizon, including both single-decision setting and multistage decision setting, and infinite time horizon. To concretize the concepts discussed, we present a simple example of Q-learning implementation for the two-stage setting using the R statistical programming language.