<p>Limited by the finite computation resources in Internet of Vehicles (IoV), it is necessary to facilitate the capacity of vehicles using computation offloading. The traditional approaches focus on maximizing utilities of computation offloading while fail to take into consideration the security of offloading strategy, which cannot be used for IoV environments directly. In addition, unsafe strategies can cause interruptions in task offloading and waste computation resources. In view of this, this paper ensures secure task offloading by mitigating interruptions caused by communication constraints such as limited transmission distance. To this end, we propose a two-layer security-aware computation offloading framework and design a multi-agent proximal policy optimization algorithm with attention mechanism (named MAPPOAM) to address the aforementioned issue. More specifically, in the feature extraction layer, the attention mechanism is utilized to obtain the relationship between the communication time and movement data of the vehicles and extract the feature of data. In the security decision layer, the agents output the offloading strategy based on the observation information. Subsequently, the algorithm in the security constraint layer analyzes the relationship between the task execution time constraints and the extracted feature to evaluate the safety of this strategy. Eventually, the agent outputs a strategy that ensures secure completion of the offloading process. The experiments with the real dataset show that MAPPOAM can effectively ensure the security of decisions compare with other baseline methods and provide valuable practical applications. Specifically, MAPPOAM outperforms the baseline algorithms, reducing both latency and energy consumption by 17.5% to 63.1%. Additionally, the security decision layer contributes to a 60.4% reduction in offloading interruptions.</p>