The effectiveness of colonoscopy in early detection of the risk for colorectal cancer depends on the endoscopist’s skill associated with Adenoma Detection Rate. To improve this rate, we propose a Deep Learning solution capable to generate masks indicating polyps in colonoscopic images. It processes colonoscopy video sequences and exploits temporal information embedded in them, using a well known technique, namely, a constrained Cross-Attention operation between current and past frames. We focused on finding a trade-off between detection performance and inference time and on assessing their mutual influence. Our model is based on PNS+, with some architectural and training process improvements. We replaced the original backbone with ConvNeXt. Also, we used Mish activations instead of PReLUs and ReLUs, Group Normalization with AdamW optimizer and cosine decay for the learning rate. These changes lead to a significant improvement of the baseline performance on the colonoscopy dataset, with a Dice coefficient value of 0.81 from 0.73, while also keeping the inference time at an adequate value (77 ms, i.e., 12 FPS) for real-time segmentation. They provide a good tradeoff between Segmentation Accuracy and processing times when compared to other state-of-the-art models. The code is available at https://github.com/UlicsDanielAlexandru/speed_perf_model or upon request from the first author.