In most fields of business, accurate predictions are the basis for future planning, whereby combining predictions of different forecasters and/or forecasting models usually generates more accurate predictions than any individual forecaster or model alone. While simply averaging forecasts using equal weights (EW) has proven to be a robust strategy in practice, an alternative approach applied in several recent papers is to learn so-called optimal weights (OW), that minimize the mean squared error (MSE) on past (training) data, and shrinking these weights towards EW. This strategy aims to learn structures from training data while mitigating overfitting and to avoid high prediction errors with novel forecasts. However, estimating OW and shrinkage levels on training samples is still subject to uncertainty and can be highly unstable especially for smaller datasets and larger sets of forecasters. This turns out to be a key problem of such approaches, which usually do not systematically beat EW approaches in practical settings. We introduce a new procedure to obtain more stable weighting schemes. The procedure learns OW on randomly drawn subsets of the training data and determines the optimal shrinkage towards EW on the respective omitted observations, resulting in varying shrunk weight vectors. Subsequently, these vectors are averaged so that the final weight each forecaster receives corresponds to the average (shrunk) optimal weight over all subsets and is asymptotically less extreme. We evaluate the procedure on synthetic datasets, where it shows benefits compared to EW as well as OW approaches in terms of the out-of-sample MSE.