This paper addresses the challenge of out-of-distribution generalization in affect modeling by introducing a novel dynamic ensemble approach. Current affect models face significant limitations in generalizability due to their reliance on the statistical learning paradigm, which assumes consistency between training and testing data distributions. However, affective data inherently violates this assumption as each user-task-annotator combination generates data from a unique distribution. We reformulate affect modeling as a multi-domain learning task and develop a methodology for constructing dynamic ensemble models that enhance generalization across different domains. Unlike static ensemble approaches, our method combines domain-specific models by dynamically weighing their contributions at the individual data point level based on distributional and prediction properties. Our approach draws inspiration from unsupervised domain adaptation techniques and determines ensemble weights proportional to the likelihood that a given data point originates from a specific domain and the prediction confidence. Experimental validation using the RECOLA dataset demonstrates that our proposed methodology significantly outperforms alternative approaches, including fixed ensemble methods and conventional unsupervised domain adaptation techniques. Notably, our dynamic ensemble method is general and can be applied to any multiple domain problem beyond affect modeling tasks, offering a versatile solution for addressing distribution shifts and reducing domain gaps in various machine learning applications.