Structure-based drug design (SBDD) accelerates drug discovery but traditionally relies on labor-intensive, simulation-based methods. Deep generative models offer a data-driven alternative, but the currently prevalent ligand-based models are often constrained by the availability of active compounds. Here, we present a new SO(3)-equivariant generative model for SBDD, using a pseudo-ligand point-cloud representations of protein cavities to optimize ligands and generate stable 3D molecules. Our model accurately models the chemical space of the protein-binding compounds. We evaluated our model on three therapeutic targets: Janus kinase 2 (JAK2), peptidylprolyl isomerase (hPin1), and Mycobacterium tuberculosis malate synthase. Our model successfully rediscovered moieties involved in key interaction with the proteins and proposed alternative moieties with bioactivity supported by the literature among the highly ranked generated samples. This work approach offers a new way to ligand optimization and drug discovery, advancing the field of public health science by enhancing the precision and efficiency of molecular design in therapeutic development.