Depression, a prevalent mental illness, often manifests itself in a set of symptoms, including subtle and sometimes subclinical abnormalities in vocal and facial expressions. However, unimodal analysis typically falls short in capturing the complexity of such signals. Multimodal learning has been the recent trend in affective computing, but many existing approaches follow independent modality-wise processing and late fusion, which limits their ability to model the intricate cross-modal interactions in a fine-grained manner. In this paper, we propose a Cross-Attention Multimodal Fusion framework for audio-visual depression detection to dynamically fuse the complementary information from speech and facial dynamics. Our model uses a two-stream encoder to first extract the audio and visual features and then aligns them using a bidirectional cross-attention mechanism. This allows the network to attend to the visual signal while processing the acoustic modality and vice versa to model the context-aware correlations related to depressive cues. We evaluate our approach on benchmark audio-visual depression datasets and show that it outperforms early- and late-fusion baselines, highlighting the potential of cross-attention based methods in multimodal affective analysis.