Single-agent reinforcement learning has been widely used to solve several traffic-related problems. These algorithms show promising results in their training environments, but their performance remains questioned when they meet other agents with the same strategy. This paper aims to show the advantages of multi-agent reinforcement learning in autonomous driving using a simple multilane highway scenario as an example. The performance of single- and multi-agent learning policies have been compared in mixed traffic. Furthermore, the effects of additional information in the observation space have also been analyzed.