Unrestricted adversarial attacks aim to fool DNNs by adding imperceptible, unconstrained perturbations to the input data. Existing methods rely on a single image modality for adversarial attacks, ignoring the role of text, which leads to low black-box transferability of fooling unseen models. To address this issue, we propose a method called TDAdv guided by specially designed prompts to improve the transferability of unrestricted adversarial examples (UAEs). Our method leverages the Stable Diffusion Model (SDM) and Visual Language Model (VLM) to generate adversarial perturbations in the latent space, and decodes the adversarial latent to obtain UAEs. First, we leverage a VLM model to recognize images and generate misleading prompts, which allows adversarial perturbations to act on features with different semantic associations. Second, two deception strategies are implemented: (1) directly perturb the image to fool the classifier, and (2) embed misleading prompts into the denoising process of SDM model and use the cross-attention mechanism to force different labels to have similar distributions in the latent space, thereby blurring the semantic boundaries between different labels. Finally, we propose a hierarchical self-attention mechanism to improve computational efficiency and preserve imperceptibility. Compared with existing methods, the proposed TDAdv and its enhanced version TDAdv-X achieve an average of 72.5% (about +7%) ASR and 80.6% (about +15%) ASR in black-box attack against four prevalent DNNs with two different architectures on ImageNet-compatible dataset.