Speaker recognition models are now widely used in daily life, with their precise recognition capabilities being applied extensively in identity recognition tasks. Recent studies have demonstrated that speaker recognition models are vulnerable to adversarial attacks, which raise significant security concerns. However, these threats have primarily been explored within the white-box attack domain, leaving a gap in the research on black-box attacks. In this paper, we propose a universal adversarial perturbation for black-box scenarios that can generate a single perturbation applicable to a wide range of attacker audio inputs, causing misclassification in speaker recognition models even with limited model information. Specifically, we employ an optimization-based method to generate adversarial examples, introducing constraints on the adversarial perturbation’s confidence and maximum distortion to balance its effectiveness and imperceptibility. Additionally, we present a novel threshold estimation algorithm to estimate the unknown model thresholds in open-set identification (OSI) tasks and speaker verification (SV) tasks. We attacked state-of-the-art models, and extensive experimental results demonstrate that our proposed black-box universal perturbation exhibits strong attack capabilities, with excellent generalization that significantly enhances attack efficiency.