Precision agriculture has been developed in recent years, leveraging different types of technologies to improve the yield and quality of the crops. Following the precision agriculture paradigm, this work focuses on the analysis of olives in images, specifically the Manzanilla cultivar. The aim is to characterize the maturity of the olives automatically in images, taking them directly in the field, while the olives are still on the tree. This methodology is a non-invasive technique that avoids the harvesting of olives from the tree to analyze them. Using the well-known maturity index and an RGB camera, the olive grove was monitored during the growing season. Thus, the images are automatically processed, identifying the olives in the images and then classifying their maturity stage. This process is carried out with an object detection model to obtain both the olive position and their maturity index. The classification problem is solved by identifying olives with a Mean Average Precision value of 0.80 at a 0.5 threshold and the metrics of the global identification of the maturity index with weighted average accuracy around 78%. Using this model, insights into the current state of maturity of the olive grove could be assessed.