Deep convolutional neural networks (CNNs) have achieved state-of-the-art performance in image classification tasks by automatically learning discriminative features from raw pixel data. However, their success often relies on large labeled training datasets and substantial computational resources, which can be limiting in resource-constrained scenarios. This study explores alternative, lightweight approaches. In particular, we compare a lightweight CNN with a combination of randomly initialized convolutional layers with an ensemble of weak learners in a stacking framework for binary image classification. This method aims to leverage the feature extraction capabilities of convolutional layers while mitigating the need for large datasets and intensive computations. Extensive experiments on seven datasets show that under resource constraints, the decision as to which model to use is not straightforward and depends on a practitioner’s prioritization of predictive performance vs. training and prediction time vs. memory requirements.