Accurate segmentation of left atrium (LA) and LA scar from the Late Gadolinium Enhancement (LGE) Cardiac Magnetic Resonance (CMR) Imaging is fundamental step in the treatment of Atrial Fibrillation (AF). However, the precise delineation of the LA and LA scars remains challenging due to the heterogeneous physiological structure, blurry boundaries and severe class imbalance. To address above problems, we propose a Boundary Loss Constrained Multi-Attention U-Net (BMAU-Net), which utilizes a three-dimensional vision Transformer module as the basic feature extraction architecture and combines the Multi-Orientation Attention Blocks (MOAB) to extract complex spatial structural information of the LA and scars. Furthermore, to address the issue of fuzzy edges, we introduce the Multi-Scale Boundary Loss Block (MSBLB) in BMAU-Net, which calculates the edge loss between features generated by the segmentation model and the edge features of the labels at different scales to obtain edge information between the LA and scars. Finally, we optimize the segmentation model by proposing the Multi-Level Parameter Sharing Pyramid Pooling Module (MPASPP) to reduce the down-sampling frequency of the model, alleviating the severity of class imbalance during feature extraction. We conduct comprehensive experiments on the LAScarQS 2022 dataset, which achieves an average Dice score of 0.778. The experimental results demonstrate that our approach achieves superior performances in comparison with state-of-the-art competitors. Our code will be released via https://github.com/Lucarqi/BMAU-Net .