For a given graph \( G \) , let \( A(G) \) , \( Q(G) \) , and \( D(G) \) denote the adjacency matrix, signless Laplacian matrix, and diagonal degree matrix of \( G \) , respectively. The \( A_\alpha (G) \) matrix, proposed by Nikiforov, is defined as \( A_\alpha (G)=\alpha D(G)+(1 - \alpha )A(G) \) , where \( \alpha \in [0,1] \) . This matrix captures the gradual transition from \( A(G) \) to \( Q(G) \) . Let \( \mathcal {G}_{n,\gamma } \) denote the family of all connected graphs with \( n \) vertices and independence number \( \gamma \) . A graph in \( \mathcal {G}_{n,\gamma } \) is referred to as an \( A_\alpha \) -minimizer graph if it achieves the minimum \( A_\alpha \) spectral radius. In this paper, we first demonstrate that the \( A_\alpha \) -minimizer graph in \( \mathcal {G}_{n,\gamma } \) must be a tree when \( \gamma \geqslant \big \lceil \frac{n}{2}\big \rceil \) , and we provide several characterizations of such \( A_\alpha \) -minimizer graphs. We then specifically characterize the \( A_\alpha \) -minimizer graphs for the case \( \gamma = \big \lceil \frac{n}{2}\big \rceil + 1 \) when \(n\geqslant 9\) . Furthermore, we obtain a structural characterization for the \( A_\alpha \) -minimizer graph when \( \gamma =n - c \) , where \( c\geqslant 4 \) is an integer.