Ball trajectory data is a crucial piece of information that must be accurately tracked for tennis professionals. High-end systems like hawk eye are crucial for precise tennis ball trajectory tracking. However, their high cost restricts their accessibility. As a more cost-effective alternative, a single camera can be used, but it poses challenges due to the tennis ball's small size and rapid movement. In this study, we propose a solution to the challenges faced by low-resource tennis analysis systems that utilize a single camera. Specifically, we introduce a heat map-based encoder–decoder network with skip connections, which learns the trajectory patterns of a tennis ball, tracks its high-speed movement, and produces dependable data. Additionally, we present a synthetic data generation technique that enhances model accuracy by a certain percentage through the generation of relevant data. The proposed model was trained using over 15 k frames and another 13 k + generated frames from various lighting conditions, camera angles, and court surfaces. The model's performance was assessed using frames captured in different conditions, yielding a precision of 98.2% and a recall of 91.4%.