The prediction of bridge structural damage constitutes a fundamental part of an efficient and anticipatory maintenance scheme. However, current prediction methodologies have been significantly suboptimal, primarily due to their susceptibility to the idiosyncrasies of individual structures and their environments. To enhance the accuracy of prediction performance, incorporating empirical damage data at the element scale is essential. Recently, the Ministry of Land, Infrastructure, Transport, and Tourism of Japan released a comprehensive nation-wide bridge database, tracking their damage progression across five-year intervals. This research initially focuses on converting schematic representations and damage data of girder bridges derived from the database into graph expression which reflects spatial relationships among elements utilizing optical character recognition techniques. Subsequently, a Graph Transformer method was established for element-level deterioration prediction. This research highlights the significance of integrating comprehensive structural data into the predictive analysis of bridge deterioration, enhanced by the application of a Graph Transformer.