In this study, we explore the limitations of early-exit architectures, which are designed to enhance computational efficiency in neural networks, focusing particularly on both single and multi-label classification tasks within the computer vision domain. We introduce a systematic evaluation framework that not only advances research in this area but also bridges an important gap in understanding how these architectures perform within the complexities of multi-label settings. Our findings reveal a significant challenge: while early-exits improve efficiency without compromising accuracy in single-label tasks, they struggle to offer similar benefits in multi-label classification, necessitating uniquely tailored strategies. Further insights from our ablation suggest that the difficulty in achieving benefits from early-exits in multi-label classification may stem from the varying complexities of processing distinct classes within a single instance. This work lays a solid foundation for future research focused on developing early-exit strategies that effectively handle the complexities of diverse classification contexts.