How can we effectively train a synthetic data generation model for vertically distributed data while preserving privacy? The key challenge in this scenario is accurately reconstructing correlations between attributes held by different parties without compromising privacy. In this study, we assume the existence of shared attributes across vertically distributed tables and explore how these shared attributes can be leveraged for improved performance. To address this, we introduce a differentially private data synthesis method called MRF-JOIN. This method joins privatized Markov Random Fields (MRFs) from different parties on the shared attributes and recovers consistency by leveraging the duplication of shared attributes across parties. Our experiments demonstrate that, compared to existing methods that do not assume shared attributes, the synthetic data generated by MRF-JOIN more effectively preserves the correlations present in the integrated data along with computational and communication efficiency, in vertically federated settings while preserving differential privacy.