One of the most profitable commercial palms in India is the arecanut. The market price of arecanuts, which is determined by the arecanuts maturity level, is the only factor that determines the profitability of arecanut plantation farmers. To reduce the financial loss of the farmer, knowing maturity level of the unharvested arecanut bunch is a tedious job due to lack of expertise. We presented an automated technique to determine the maturity level of the unharvested arecanut bunches in order to address this problem. The proposed method works on two phases: first, an optimized U-Net model was used for semantic segmentation of the arecanut bunch images then the classification of the segmented arecanut bunches was performed using transfer learning approach. Experiments were conducted on RGB and saturation channel of HSV color space images, and a comparative study of RGB and saturation channel of HSV color space is presented. Result of the experiments show that the segmentation and classification of the arecanut bunches from input image is efficient using RGB color space.