Community detection, while crucial for analyzing complex networks, faces vulnerabilities from adversarial attacks that exploit fixed heuristics for edge manipulation, lacking adaptation to intrinsic structural dynamics. In this paper, we propose the Modularity-Guided Adversarial Matrix Factorization (MGAMF) framework, which integrates modularity inversion guidance with incoherent embedding regularization to establish a learnable and adaptive attack mechanism. Specifically, our approach jointly learns node embeddings for both the original and perturbed networks, thereby capturing a more comprehensive representation of their respective structures. We propose a modularity inversion strategy that reinterprets modularity as an adversarial signal, effectively misleading detection processes. Furthermore, by imposing incoherent regularization on the Gram matrices of the learned embeddings, we enforce a structural disparity between the original and perturbed embedding spaces, enabling the perturbation matrix to more accurately pinpoint adversarial vulnerabilities in community-critical substructures. Extensive experiments on benchmark datasets demonstrate that MGAMF outperforms existing attack methods in both effectiveness and computational efficiency, and it exhibits robustness across various community detection algorithms (e.g., Fastgreedy, Louvain, Infomap, and Walktrap).