Accurate yield prediction allows farmers to strengthen their relationships with their customers, as they are able to guarantee supply and deliver contracts on time. Neural networks have made it possible to build robust and accurate prediction models. However, Neural Nets are quite demanding in terms of quality training datasets. In addition, their black-box nature makes the interpretation of parameters complex. To address these problems, we proposed an FCN network to construct a trainable yield prediction model. To take the full advantage of the training, we plug a trainable function to calculate tree heights as a layer. Such a layer takes an RGB image of a tree as input and produces its height. The output of such a layer is used in a linear model to estimate the tree’s yield. This report is supervised by a guidance information built from historical data. The proposed approach has been evaluated on data collected from a UAV on a cashew field and showed promising results.