Deploying deep learning models for Sleep Apnea (SA) detection on wearable devices remains challenging due to stringent energy and computational constraints. Recent studies have predominantly focused on either complex reprocessing techniques suitable only for high end GPUs or utilized aggressive model compression (e.g., binarization) that significantly degrades diagnostic sensitivity. To address these limitations, this paper proposes an optimized hardware software co-design framework built upon a customized RISC-V processor platform. Unlike prior approaches that rely on raw 1D signal processing, we introduce a Lightweight Spectrogram Transformation strategy (30 \(\times \) 30) to explicitly capture the time-frequency spectral anomalies associated with apnea events. To support this, we optimize the RV-SCNN hardware architecture by removing redundant Spiking Neural Network (SNN) components, dedicating resource utilization exclusively to CNN acceleration. The system is implemented using 8-bit integer (INT8) quantization with a dynamic bit shifting mechanism. Experimental results on the standard PhysioNet Apnea-ECG dataset demonstrate the robustness of the proposed co-design. The software baseline (32-bit Floating Point) achieves an accuracy of 95.08% and a sensitivity of 96.12%. When deployed on the optimized hardware accelerator using INT8 quantization, the system maintains an accuracy of 94.54% and a sensitivity of 95.80%, exhibiting a negligible performance degradation of less than 0.6%. Furthermore, the optimized accelerator delivers a 9.86x speedup over a standard RISC-V core, reducing inference latency to 7.16 ms, thereby providing a superior trade-off between diagnostic precision and hardware complexity for next generation wearable healthcare devices.