This paper proposes a multi-modal model, namely MMF2Drug, fusing the sequences and structures of proteins, physicochemical properties and structures of molecules, to generate high-quality molecules. MMF2Drug includes a Feature Extraction (FE) module, a Multi-Modal Fusion (MMF) module and a Conditional Molecule Generation (CMG) module. FE module is used to extract the different modal embeddings from proteins and molecules. MMF module aims to capture the consistency among various modalities by Iterative Multi-scale Channel Attention (IMsCA), which can assist the model in identifying key feature channels that share commonalities across different modalities, enhancing the alignment of features. CMG module combined with Generative Adversarial Network (GAN) can significantly improve capabilities in exploring molecular chemical space, generating molecules with novel structures. It uses the protein features as constraints to more accurately generate molecules for specific proteins. Experiments show that MMF2Drug has better performance for generating molecules. Case studies show that targeted drug molecules generated by MMF2Drug exhibit strong affinity for two key targets (KRAS and EGFR) in pancreatic cancer.