The work presented in the manuscript introduces a single-phase ego-vision algorithm in unconstrained and unstructured environments for hand detection. We aim to built an automated system for detecting hands from images using mostly wearable cameras. We use a state-of-the-art object detection algorithm RetinaNet to detect hands. In experiments, we evaluate hand detection algorithm in a laboratory as well as real-world scenarios. We achieve improved performance on widely utilized benchmark datasets in current literature for hand detection algorithms. The best average precision (AP) our proposed hand detection system achieved is 0.83 on Oxford hand, 0.94 on VIVA hand, 0.70 for TV Hand, and 0.90 on the 100DOH datasets. The system’s ability to generalize across different datasets for hand detection has also been validated.