Many current topology structural optimization schemes have been verified using simple 2D spatial trusses or 3D moment frames as illustrative examples. However, these methods are not sufficiently practicable, probably resulting in unsatisfactory optimization results for real high-rise building structures. Initially, the applicability and computational efficiency of the standard whale optimization algorithm (WOA) are improved by introducing several measures, including the inertial weight, a method for mapping variables, and an upper-bound algorithm. Then a novel computationally efficient framework for the optimization design of high-rise buildings with complex structural system is proposed in this study based on the improved version of WOA. Shear wall layout and location of outrigger and belt-truss system are both considered as discrete design variables. The optimization algorithm is combined with conceptual design rules, including vertical continuity and plan symmetry of the shear wall layout design, to guide the algorithm search direction. A case study is presented demonstrating the potential of the proposed framework.