<p>Current motion controllers for table tennis agents in virtual reality primarily rely on the opponent’s ball properties during an ongoing rally. However, despite leveraging visual reinforcement learning, these controllers still face difficulties in executing complex strategies, such as alternating the ball placement between the left and right edges of the table. This strategy is effective because it disrupts the opponent’s rhythm, forcing them to adjust their positioning and pace, which enhances its utility in both real-world table tennis training and matches. While the vanilla transformer excels at capturing correlations between distant local tensors, the agent’s observations—such as pose, ball velocity, and landing point—are highly discrete, making it difficult to accurately infer the opponent’s intent. To address this challenge, we propose the Opponent Temporal Goal-Aware Transformer (TOGA-Net). TOGA-Net first predicts potential goals in the opponent’s sequence using a temporal goal-aware adversarial discriminator–generator transformer module. Based on these predicted goals, it then adjusts the agent’s response strategy. Finally, the high-level adjustments are broken down into specific positional and rotational trajectories for each joint, resulting in a refined policy. We validate TOGA-Net on the OpenTTGames and Google table tennis robot datasets, where it outperforms recent state-of-the-art methods.</p>