Many constrained optimization problems, including those relevant to infrastructure planning, e.g., energy systems or logistics, can be effectively solved using white-box solvers based on linear programming. While these algorithms are well understood by the experts who developed them, explanations of the solutions they find are still necessary to communicate their implications to laypeople. However, it is unclear what such explanations should look like since the linear program solvers’ high-dimensional and abstract representations of the problem likely do not match human representations. Here, we propose an algorithm for finding rationales that align with human representations of constrained optimization problems. The proposed algorithm incorporates key insights from prior research on the structure, complexity, and representations of human explanations for constrained optimization. Specifically, we introduce a grammar of predicates derived directly from participants’ explanations and behavioral data from our previous studies on human optimization strategies. With a prior that regularizes rule complexity, this grammar forms the foundation of a rational rules model, which we use to generate rationales modeled after human explanations. Given that human explanations for constrained optimization problems reflect a sequential decision process, our approach searches the space of sequential solution representations within a Markov Decision Process to identify the most interpretable sequence and corresponding rationale. We evaluate our algorithm on human solutions and demonstrate that the generated rationales have a high dataset description score and complexity similar to human explanations, suggesting that the rationales capture human decision processes well and, therefore, align with the representations and structure of human explanations.