<p>Traditional methods approach the speech enhancement task through signal processing techniques and certain statistical assumptions. They primarily assume that noise signals are stationary. However, because noise in natural environments is often non-stationary, traditional methods face limitations and tend to perform poorly in non-stationary environments. Deep neural networks have driven significant advancements in speech enhancement. However, the improvement in speech quality remains limited due to existing techniques' challenges in effectively capturing contextual information across various scales. The fully connected layers in CNNs, in particular, have difficulty modeling temporal structures accurately., while the Redundant Convolutional Encoder-Decoder (RCED) cannot explicitly extract speech signals, limiting its representational capacity. To address this, we introduce an attention mechanism to focus on the most valuable information—specifically, feature maps containing speech. Additionally, implementing LSTM often involves substantial computational overhead due to the need to calculate input, output, forget gates, and cell memory. This can pose challenges when deploying models on resource-limited devices.Most research has focused on finding an optimal speech feature to extract hidden local features while ignoring the global relationships of the speech signal. In this paper, we propose a method that utilizes a multiscale feature extraction (MSFE) structure with global and local information to obtain comprehensive speech features. Time–frequency distributions of spectral components play a critical role in speech enhancement, yet they are often underexplored in research. To address this gap, we propose a novel model, Time Frequency Attention D3Net Multi-Scale U-Net (TFAD3MUNet), which integrates a nested U-Net architecture with time–frequency attention (TFA) and D3Net. Our model uses an encoder-decoder structure with skip connections to promote effective information flow. Within this framework, we incorporate a dilated Dense Net (D3Net) block and a multi-scale feature extraction block (MSFEB), enabling the capture of extensive contextual information across both local and global features, thus enhancing speech reconstruction. D3Net also resolves aliasing issues associated with dilated convolutions in Dense Net models. The time–frequency attention mechanism further improves the model by identifying the most critical channels, frequencies, and temporal features for speech enhancement. By utilizing position information, this mechanism generates a 2D attention map is used to highlight the key time–frequency distribution of speech through dual processing branches: time-frame attention and frequency attention in parallel. D3Net introduces an innovative multi-dilated convolution technique, simulating multiple resolutions simultaneously by applying variable dilation factors. Our model's time–frequency attention mechanism integrates both spatial and channel domains. It initially prioritizes feature channels, then refines features across frequency and time dimensions, effectively minimizing information loss and enhancing the model's essential characteristics.Additionally, the time–frequency attention module supports module testing and allows flexible kernel size selection, enhancing the network's performance. The integration of D3Net and TFA modules strengthens the model’s ability to extract features and aggregate contextual information at the utterance level. This proposed model demonstrates superior performance in terms of short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ) when compared to baseline methods.The proposed model increases the average PESQ and STOI scores by 35.9% and 18.1%, respectively, for the Common voice dataset, and by 35.2% and 4%, respectively, for the LibriSpeech dataset compared to noisy speech.</p>

Speech enhancement using nested U-net with time frequency attention and D3 net

期刊论文
发表日期：

2025年4月12日

查看全文

Sunnydayal V,
Raparla Sindhu

摘要

Traditional methods approach the speech enhancement task through signal processing techniques and certain statistical assumptions. They primarily assume that noise signals are stationary. However, because noise in natural environments is often non-stationary, traditional methods face limitations and tend to perform poorly in non-stationary environments. Deep neural networks have driven significant advancements in speech enhancement. However, the improvement in speech quality remains limited due to existing techniques' challenges in effectively capturing contextual information across various scales. The fully connected layers in CNNs, in particular, have difficulty modeling temporal structures accurately., while the Redundant Convolutional Encoder-Decoder (RCED) cannot explicitly extract speech signals, limiting its representational capacity. To address this, we introduce an attention mechanism to focus on the most valuable information—specifically, feature maps containing speech. Additionally, implementing LSTM often involves substantial computational overhead due to the need to calculate input, output, forget gates, and cell memory. This can pose challenges when deploying models on resource-limited devices.Most research has focused on finding an optimal speech feature to extract hidden local features while ignoring the global relationships of the speech signal. In this paper, we propose a method that utilizes a multiscale feature extraction (MSFE) structure with global and local information to obtain comprehensive speech features. Time–frequency distributions of spectral components play a critical role in speech enhancement, yet they are often underexplored in research. To address this gap, we propose a novel model, Time Frequency Attention D3Net Multi-Scale U-Net (TFAD3MUNet), which integrates a nested U-Net architecture with time–frequency attention (TFA) and D3Net. Our model uses an encoder-decoder structure with skip connections to promote effective information flow. Within this framework, we incorporate a dilated Dense Net (D3Net) block and a multi-scale feature extraction block (MSFEB), enabling the capture of extensive contextual information across both local and global features, thus enhancing speech reconstruction. D3Net also resolves aliasing issues associated with dilated convolutions in Dense Net models. The time–frequency attention mechanism further improves the model by identifying the most critical channels, frequencies, and temporal features for speech enhancement. By utilizing position information, this mechanism generates a 2D attention map is used to highlight the key time–frequency distribution of speech through dual processing branches: time-frame attention and frequency attention in parallel. D3Net introduces an innovative multi-dilated convolution technique, simulating multiple resolutions simultaneously by applying variable dilation factors. Our model's time–frequency attention mechanism integrates both spatial and channel domains. It initially prioritizes feature channels, then refines features across frequency and time dimensions, effectively minimizing information loss and enhancing the model's essential characteristics.Additionally, the time–frequency attention module supports module testing and allows flexible kernel size selection, enhancing the network's performance. The integration of D3Net and TFA modules strengthens the model’s ability to extract features and aggregate contextual information at the utterance level. This proposed model demonstrates superior performance in terms of short-time objective intelligibility (STOI) and perceptual evaluation of speech quality (PESQ) when compared to baseline methods.The proposed model increases the average PESQ and STOI scores by 35.9% and 18.1%, respectively, for the Common voice dataset, and by 35.2% and 4%, respectively, for the LibriSpeech dataset compared to noisy speech.