Cardiovascular diseases are a leading cause of mortality worldwide, emphasizing the urgent need for early and accurate diagnosis. Traditional diagnostic techniques often rely on manual assessments by specialists, which are time-consuming, subjective, and resource-intensive. This study introduces SWASSA, a framework designed to classify cardiovascular conditions based on heartbeat sounds into normal and abnormal categories, aiming to improve the early detection of heart disorders. A multi-modal heartbeat sound dataset is utilized to evaluate various models. To ensure robust feature extraction from the heartbeat sound data, advanced signal processing techniques are employed, including spectral analysis, spectrograms, and Mel-Frequency Cepstral Coefficients (MFCCs). The framework explores a range of algorithms, including traditional multi-modal approaches and hybrid models, leveraging the feature extraction capabilities of CNNs, LSTMs, and their combinations. By automating the classification process, SWASSA aims to make cardiovascular screening more accessible, efficient, and reliable. Through experimental analysis, the model’s performance is evaluated using various metrics, with results demonstrating that SWASSA outperforms baseline models, achieving a precision of 95%, a recall of 95%, and an F1-score of 95%. These findings suggest that SWASSA significantly contributes to the development of intelligent, non-invasive diagnostic tools suitable for large-scale deployment in clinical settings.