Machine translation training with reinforcement learning has shown improvement in the translation quality. In previous works, the BLEU score was used as a reward function for reinforcement learning being the metric the most widely employed for evaluating machine translation models. Nevertheless, BLEU is not the only metric available to evaluate machine translation models. In this work, we use four different measures as reward functions, and we study their impact on the translation quality. We use these rewards to train a transformer model with reinforcement learning to perform translation between Arabic and English. Experimental results with the IWSLT dataset reveal that the GLEU score enhances the Arabic-to-English translation quality and the BLEU score improves the English-to-Arabic one.