<p>Home security through traditional mechanisms suffers from several limitations where, mechanical systems are vulnerable to assaults. Therefore, computer vision-based smart surveillance is a more secured method of physical access to the homes. Intelligent human authentication to monitor the entry points of the smart homes is possible through video analysis. In this paper, we present a novel technique that is based on face recognition assisted human detection through closed circuit television camera (CCTV) videos to enter in a house. Our proposed method utilized haar cascade face detection technique to detect authorized human to offer physical access to the smart home. The base model is learned with a pre-trained fine-tuned deep convolutional neural network (DCNN) for face detection in the video frames. Besides, a self-created dataset of three human has been used to efficiently train the two popular fine-tuned models i.e., Residual network-50 (ResNet-50) and MobileNetV3. The presented approach shows promising performance for authorized physical access to the smart homes. The overall performance of our technique in terms of accuracy is 99.96 and 99.87% respectively with ResNet50 and MobileNetV3 in seen environment. Whereas, it shows an accuracy of 96.3782 and 95.234% respectively for both the employed models in unseen environment that exhibits superiority as compared to traditional mechanism of home surveillance. However, MobileNetV3 outperform ResNet50 in terms of low latency due to its lightweight size. In addition, the extensive experimentations with ablation study and error analysis confirmed that proposed model yield promising performance for home surveillance via video analysis.</p>