Novelty Detection tackles the problem of identifying samples that do not belong to classes (id) observed by a given model during its training phase. In particular, Object-based Novelty Detection (OND) introduces an additional granularity compared to image-level novelty detection, allowing to process multiple objects at the same time. The majority of novelty detection approaches either assume full control over the initial (pre)training phase, or solely focus on post-processing of the pretrained model outputs at inference time. As a consequence, these solutions inevitably discard the precious knowledge within out-of-distribution (ood) data encountered after deployment. In this work, we assume that feedback in form of cheap annotations, confirming or rejecting the model’s novelty decisions on detected objects, can be incrementally obtained. We then investigate how incorporating this feedback can strengthen models robustness. We propose both \(\textcircled {1}\) a  novel setting and \(\textcircled {2}\) benchmark for OND with incremental feedback loop, focusing on the continuous incorporation of ood information over the model’s life-cycle. Furthermore, we propose \(\textcircled {3}\)  a new method to effectively incorporate such feedback. Our approach, named iConP, consists of a lightweight novelty detection module, optimized continuously with the received feedback. Our method reduces the False Positive Rate (FPR) of the object detection model by \(\sim \) 64%, when leveraging the continuously received feedback, while maintaining the object detection performance on known classes unaltered.