Plastic waste has become a major environmental hazard for the past few decades. Automated waste segregation using Deep learning techniques has proven to be a very efficient way to identify and segregate the generated waste. Traditional and CNN-based machine learning models developed for plastic waste classification had pre-defined labels and static classes associated with them. In recent times, due to technological improvement in production methodology, different categories of plastic waste have been generated into the environment. A plastic-type identified under a particular category previously is now classified under a different category based on the recycling technology changes, property character, and its impact on the environment. To cater to this current need, the zero-shot open vocabulary capabilities of Vision Language Models (VLMs) can be used to identify unseen waste types or novel classes. However, the real-time zero-shot object detection on resource-constrained edge devices such as microprocessors, mobile devices, etc. poses several challenges such as memory constraints, computational limitations, and heterogenous hardware configurations. This paper provides a comprehensive understanding of the optimization techniques that can be employed in overcoming the shortcomings of edge deployment of zero-shot open-vocabulary object detection models and how to use them effectively for plastic waste classification.