A method for automated petrographic analysis of natural aggregates using hyperspectral and colour (RGB) images and AI algorithms in the form of deep learning networks is presented. Certain constituents of aggregates can negatively affect the functional properties of the produced concrete and cause damage such as alkali-silica reaction or other damage patterns. The aim of the research was to reliably and automatically classify a selection of aggregates used in concrete production based on their spectral properties using RGB and hyperspectral images. For this purpose, RGB images were acquired using VIS line scan cameras and hyperspectral images of various natural aggregates were acquired using a hyperspectral camera in a selected spectral wavelength range from 960 to 1700 nm and analysed using segmentation, dimension reduction (multivariate data analysis methods) and AI application. A convolutional network (CNN) were used to classify the dimensionally reduced hyperspectral images and RGB images. As a result, a mean detection rate of 89–98% was achieved for classification into the three upper classes ‘non-critical’, ‘critical I’ and ‘critical II’. At the same time, concrete durability tests were carried out to investigate the damage potential of selected critical aggregates in concrete.