Ultrasound is a widely used diagnostic modality that generates substantial amounts of ultrasonic medical data. With the rapid development of deep learning and effective utilization of medical data, computer-aided intelligent diagnosis methods have gained prevalence in recent years, particularly in the classification of benign and malignant lesions. However, relying solely on coarse-grained classifications might not provide sufficient assistance to sonographers. In this study, we construct two types of fine-grained ultrasound datasets (breast and gastroscope) with pathological proof and propose an Approximated Kronecker Product (AKP) with a Hollowing Hash Table (HHT). By adopting our proposed methods, we can effectively utilize higher-order correlations between channels and enhance performance on multiple datasets. Additionally, we introduce the concept of Covariance Matrix Equivalent Grayscale Image (CMEGI) which well combines fine-grained classification with radiomics. The framework we designed helps to overcome the problems of inexplicability and the high cost of data, thereby facilitating the extraction of interpretable rules from low-cost medical images without needing heavy annotations.