Yield prediction is the process to predict the amount and size of crop/fruit production. The prediction can be made from different aspects, e.g., soil condition, crop management, weather record/prediction, etc. Some of the data, such as soil condition, is hard to obtain because the onsite collection is expensive and time-consuming. Weather data, on the other hand, is more accessible. Multiple challenges are present in yield prediction: 1) achieving multiple objects(e.g. amount and size prediction) can be hard because of the overfilling tendency of the neural network; 2) uncertainty, which is important to convey the confidence of the machine learning prediction, is not modeled in current baseline models. To address these challenges, this article proposes an innovative framework for the prediction of fruit yield with the ability to estimate both the count and the size of fruits. The correlation of count and size is characterized and utilized with a Multi-Task Learning (MTL) framework. A transformer is used for feature transformation to help the recognition of the temporal dependencies in the input data. Furthermore, this paper also proposes the uncertainty quantification for MTL which is neglected in previous yield predictions. Extensive experiments with the kiwifruit yield in New Zealand are conducted to illustrate the effectiveness of the prediction model and the successful quantification of uncertainty.