The impressive capabilities of Large Language Models (LLMs) enable them to perform recommendation through prompting, facilitating a novel paradigm of universal recommender systems. However, in practice, LLMs often exhibit some inherent stereotypes that should be avoided in recommendations. This necessitates aligning LLMs to meet the fairness requirements of recommendation systems. But the typical alignment methods often require substantial human labors for external supervision, which is further exacerbated when addressing fairness across multiple sensitive attributes. To address this limitation, we propose a novel Mixture of Experts (MoE) contrastive learning approach to enhance fairness of LLM-based recommenders without additional external supervision. Specifically, we first leverage contrastive learning, along with counterfactual data augmentation, to improve fairness by reducing the difference between the hidden states of contrastive sample pairs. Besides, to better handle scenarios involving multiple sensitive attributes, we propose a LoRA-based MoE framework to disentangle attribute relationships for efficient fine-tuning. And to avoid the distortion from the varying sample training difficulty due to the differing involved attributes, we further incorporate a tailored Curriculum Learning strategy, which progressively trains on samples of increasing difficulty based on the sensitive attributes involved. Finally, extensive experiments on two public datasets demonstrate the effectiveness of our proposed method.