Multi-agent reinforcement learning (MARL) has shown great promise in tackling complex tasks involving multiple interacting agents. However, traditional MARL approaches often face difficulties in efficiently capturing inter-agent dependencies and coordinating actions over long-term horizons. To address these limitations, a novel approach multi-head attention-based multi-agent deep reinforcement learning (MHAMADRL) is introduced that integrates multi-head attention mechanisms with transformer networks, into the MARL framework. By utilizing the multi-head attention mechanism, our method enables each agent to dynamically attend to and prioritize information from various parts of the environment and other agents, thereby enhancing both individual and collective decision-making processes. Simulation results in different challenging multi-agent environments show that our proposed method achieves a V2I sumrate of 67 Mb/s and a V2V link success probability of 0.998, highlighting the effectiveness of using a transformers network for resource allocation in V2X environment.