During the COVID-19 pandemic, forecasts of disease incidence and burden were used by policy-makers around the world to assist in decision-making. These forecasts were made by various models and were often combined into an ensemble to produce an average forecast across several models. These ensemble forecasts are often perceived to be more trustworthy than the forecast from a single model. However, not all models produce good forecasts during all times in the pandemic, and the unweighted averages used to form ensemble forecasts will, therefore, not be optimal. How can we better weigh individual models in an ensemble? When can we trust individual models, and when can we trust the output of the overall ensemble? This perspective examines COVID-19 ensemble forecasts and the challenges that need to be solved before the next pandemic.