Bridges play a crucial role in highway infrastructure systems. Driven by climate change and anthropogenic factors, bridge condition will deteriorate and increase the risk of failure. However, current bridge deterioration models primarily focussed on homogeneous deterioration modelling and could not correctly capture the spatial and temporal uncertainty of infrastructure assets. Bridges in the real world differ significantly and they must be treated as a heterogeneous system in deterioration modelling, which is lacking. To bridge this gap, this paper proposes a stochastic deterioration modelling approach. The infrastructure assets are treated as a heterogeneous system, and the deterioration is modelled as a continuous-time continuous-state non-stationary gamma process. Historical data of more than 5000 bridges in Ontario from 2000 to 2020 were used to demonstrate the proposed modelling approach. Various key bridge attribute parameters are considered and then formulated into the shape function of gamma process model to capture the heterogeneity of bridge deterioration. Then, the maximum-likelihood estimation approach is employed for parameter estimation, and the Akaike information criteria is employed to select the best model structure. Based on the modelling results, the most influential parameters that affect the deterioration are also identified, bridge materials and geographical region, number of spans have significant influence on bridge deterioration, whereas the traffic volume, structure type are of secondary importance. This study will help bridge owners and highway agencies model and predict bridge deterioration and make efficient inspection and maintenance decision-making.