<p>Multi-objective reinforcement learning (MORL) algorithms predominantly rely on scalarization functions parameterized with the preferences of the decision maker to derive trade-off solutions. However, this is not always feasible or desirable in the deterministic settings where scalarization functions are hard to specify, or where Pareto optimal solutions vary solely due to changes in the multi-objective reward function. Therefore, we consider a goal-augmented dynamic multi-objective Markov decision process (GA-DMOMDP), which enables the learning of Pareto optimal solutions through specifying and pursuing appropriate goals rather than relying on explicit scalarization functions. Restricted to the above GA-DMOMDPs, a multi-objective goal-oriented reinforcement learning (MOGORL) algorithm is further proposed so that the possibly changing Pareto optimal solutions can be tracked. In our algorithm, an on-line learning mode is proposed to continuously detect new goals, and to simultaneously pursue different goals by a hindsight relabeling strategy. Experimental results show that our algorithm can learn the Pareto optimal solutions in the deterministic environments with either static or dynamically changing rewards, regardless of the shape of Pareto optimal fronts, which outperforms generalized MORL algorithms with linear and Chebyshev scalarization functions.</p>