Handwritten signatures remain a primary method for identity verification due to their accessibility and non-intrusiveness. Despite advances in deep learning and computer vision, signature recognition still faces challenges, particularly in handling small-sample training and high intra-class variability. To address these issues, we propose the first audio-visual multimodal framework for handwritten signature recognition. Our model, MSSSRNet, integrates a residual module-based two-path fusion network and an attentional feature fusion (AFF) module. By employing a cross-modal audio-visual fusion strategy, the network captures both global and local features from signature images and pen stroke audio, learning their interactions through an attention mechanism. To the best of our knowledge, this is the first application of audio-visual multimodal representation in handwritten signature recognition. To support this research, we introduce a novel dataset comprising 2,000 real samples from 100 individuals, the first publicly available dataset to include both signature images and pen stroke audio signals. Experimental evaluations on our dataset and two widely used public benchmarks demonstrate the effectiveness of our approach, consistently outperforming state-of-the-art unimodal methods. Our findings highlight the biological significance of pen stroke audio in signature verification. The dataset is available at https://github.com/helloGit12345/MSSSR .