Image segmentation is helpful for a variety of tasks in many fields, including image analysis, object recognition and characterization, content-based image recovery, and foreground detection. Several models have been developed by considering spatial analysis and adjustment of skin models to continuously enhance the segmentation process. However, the field of hand image segmentation using skin identification is still controversial due to high pixel variation and particularly low human skin shading and thus, limits the performance of models. This research designs a HySeg-Net framework, by proposing marker-based Automated Maximal Similarity-Based Region Merging (AMSRM) that enhances the segmentation for recognition of foreground objects. The approach also leverages stoppage limit, region merging, and convolutional neural networks (CNN) techniques to make the framework robust for skin data belonging to different geographical regions. The proposed technique is validated on two standard datasets, namely American Sign Language and Polish Sign Language with varying backgrounds. The framework achieves significant improvements in various parameters including Probability Random Index (PRI), Global Consistent Error (GCE), Variation of Information (VOI), Mean Square Error (MSE), and subsequently peak signal-to-noise ratio (PSNR). Ablation study demonstrates that the results of HySeg-Net are better as compared to the state-of-the-art techniques.