A major threat to cybersecurity is the early detection of zero day malware images which remain almost invisible due to their unique nature. We propose a new way that uses a MobileNetV2 architecture combined with open-set classification to tackle this problem. MobileNetV2 model takes malware images as input and learns complex visual features that separate malicious from benign samples even for zero-day malware. Open-set classification improves this power such that it can differentiate between instances not belonging to the trained classes, allowing the model to generalize well on novel unseen malware families. Our work emphasizes relevance of open-set classification in zero-day malware detection that can lead to better secure and proactive mechanism against cyber threats. State-of-the-art testing on benchmark datasets such as Malimg and Malevis demonstrates the framework’s superior performance in a number of classifications settings: binary-class, multi-class and open-set problems. MobileNetV2 not only excels on the benchmark datasets, but actually outperforms current state-of-the-art approaches. MobileNetV2 regularly scores \(99.3\%\) and \(98.4\%\) for binary and multi-class classifications. As the capability of the framework can also be used to detect unknown malware, we can observe that \(92.3\%\) accuracy in Malimg or \(94.8\%\) Malevis.