Deep neural networks are vulnerable to adversarial attacks, as attackers can generate effective adversarial examples by adding imperceptible perturbations to inputs without any access to the training process. Although ensemble defense has been shown to be effective in defending against adversarial attacks by combining diverse models, promoting adversarial robustness while maintaining generalization accuracy remains a significant challenge. We propose an effective ensemble training strategy, Transferability Reduced via Orthogonal Decomposition (TROD), to train a robust ensemble with low transferability by encouraging differentiated data learning among base models. Specifically, during the training of the ensemble model, the input data is decomposed into mutually orthogonal subspaces, which are used for the training of different sub-models. The orthogonality of the learned representations maximizes the divergence in feature understanding across sub-models, making adversarial examples crafted against one sub-model less transferable to the others. The experimental results indicate that this diversification of feature understanding effectively reduces the attack transferability of adversarial examples under white-box settings.