Graph-based deep learning provides an efficient, single-stage solution for reconstructing digital city models from Very-High Resolution (VHR) aerial or satellite imagery. However, these models are highly sensitive to label quality—particularly in detecting roof corners and identifying the relationships between points to form polygons.—making them vulnerable to the misaligned annotations often found in open-source datasets. This study investigates the impact of annotation quality on reconstruction performance and proposes a mitigation strategy using generative synthetic datasets. Leveraging Stable Diffusion with ControlNet, we generate synthetic Red Green Blue (RGB)–label pairs conditioned on misaligned label masks. This process realigns the imagery to noisy annotations, producing spatially consistent training data. Texture control is introduced to enhance visual fidelity and support better feature learning. We evaluate this approach using PolyRoof, a graph-based reconstruction model. While models trained solely on synthetic data achieve moderate performance, pretraining with synthetic datasets followed by fine-tuning with accurately labeled real data leads to significant gains. When integrated into the training pipeline, synthetic pretraining improves reconstruction quality and can even outperform models trained exclusively on real labels. These improvements are observed across both quantitative metrics and qualitative visual results. Overall, the findings indicate that synthetic data—when spatially aligned with noisy annotations and combined with transfer learning—serves as an effective pretraining resource. This can be followed by fine-tuning on real datasets as part of a dataset enrichment strategy aimed at addressing label quality issues in graph-based building reconstruction.