Semantically Controlled Texture Synthesis by Diffusion Model

Texture synthesis is a versatile task applicable across various domains, from entertainment to medical imaging. Traditionally, texture synthesis involves generating one type of texture per image. However, real-world objects and scenes often comprise multiple materials. Nowadays, image synthesis focuses on solutions based on generative neural networks. One of the possible controlling mechanisms for such methods uses an input semantic mask that specifies the layout of individual objects. This approach can also be beneficial for synthesizing combined textures without needing to concatenate regions of different textures by additional approaches. In this study, we propose a method for semantically controlled texture synthesis by diffusion model. Used architecture of the proposed method is based on the adapted solution for semantic image synthesis, which we tuned to improve the quality for the texture synthesis task and faster sampling/training. The proposed method also contains a specialized training approach to reduce the sharp seams at the edges of different regions for multi-texture semantic synthesis on a dataset containing only single-region texture images. Additionally, we experimented with an approach for the synthesis of texture transition/interpolation during the synthesis process. Evaluation of our proposed method shows improved quality of texture synthesis compared to the existing semantically controlled synthesis solutions.