In this paper, we investigate randomly terminating two-player zero-sum Markov games. This game model differs from infinite-horizon discounted zero-sum Markov games and can be used to model many practical scenarios; however, related work on such games is still insufficient. We propose the regularized minimax-V learning algorithm and prove that the value sequence generated by this algorithm converges to the minimax value function with an appropriate choice of regularization parameter sequence. This algorithm applies the Euclidean regularization technique to accelerate the convergence in a different way compared with previous literature. Through simulation experiments, we demonstrate the convergence of regularized minimax-V learning algorithm in the ratio game and a randomly generated Markov game. To the best of our knowledge, for randomly terminating two-player zero-sum Markov games, this paper presents the first accelerated NE-solving algorithm.