This paper introduces an automatic human authentication system for mobile and handheld devices, utilizing photo of the palm area termed as palm-photo. A novel regularization technique has been proposed for the deep learning architecture, which incorporates a modified softmax loss function and maximizes the mutual information between input samples and their learned embeddings. This regularization technique effectively enables the extraction of highly discriminative features, even when dealing with small-sized Region of Interest (RoI) samples. The palm-photo samples are collected via a touchless application that captures video using the mobile device’s rear camera, subject to varying lighting and orientation. The system selects the sharpest frame through successive blurring, performs RoI segmentation and pre-processing, and learns feature vectors using the proposed network and loss function. Experimental results demonstrate exceptional performance, achieving an EER of 0.002% and CRR of 100% even with small RoI images ( \(50\times 50\) ). This approach outperforms existing state-of-the-art systems in terms of accuracy and rotation/translation invariance. The proposed system offers a reliable method for device unlocking and authentication on mobile and handheld devices.