Cross-domain image style transfer task is an attractive topic for several applications, such as image-to-image style transfer, text-to-image style transfer, artistic image generation, etc. In cross-domain image style transfer tasks (e.g., image-to-image style transfer, artistic image-to-image style transfer, text-to-image style transfer, etc.), training becomes cumbersome due to differences in data distribution across domains and complex model architectures. Unlike existing domain adaptation and domain-independent methods that focus on robust and sufficient feature extraction, this work focuses on disentangling the latent space through latent optimization. For this purpose here we propose a new idea of styled image generation from the latent space of StyleGAN which works well for image-to-image and text-to-image style transfer. We critically analyzed the low-dimensional latent structure and its effect on cross-domain image style transfer tasks and finally proposed a method along with a latent optimizing procedure to overcome the problem of style transfer. The experimental results on different standard datasets show that the proposed model is robust, effective, and generic compared to the state-of-the-art models.