Wheat ranks with most important cereal crops as a worldwide food for millions of people, by its adaptability to be grown in various climates and soil types. Enhancing Wheat productivity presents a global issue. Stripe rust pathology, known as Yellow Rust, is one of the most damaging Wheat diseases. Early diagnosis of Wheat leaf along with estimating the severity level enables much more preciseness in disease monitoring. In this work, a severity level estimation system is proposed by learning deep features dissimilarities derived from several CNN models. In a first step, VGG16, MobileNetV2 and a customized CNN model are trained to generate images deep features. Then, a feature selection process for data consolidation, is fed after that into a dissimilarity measure phase. Both a SVM and an ANN algorithms are trained to make the final decision from an inter and intra dissimilarity classes. Experiments are conducted on Yellow-Rust-19 dataset, which contains 15000 Wheat leaf images. Results obtained reveal that the proposed system gives an improvement around 2.4% in the overall classification accuracy compared to individual CNN models.