The optimization of well controls and perforation locations constitutes an essential procedure in geological carbon sequestration (GCS). Such optimization tasks can be computationally intensive due to the need for many forward simulations. This paper presents a novel transformer-based deep learning neural network as a surrogate model to accelerate the optimization process. Transformer layers are utilized to efficiently capture the temporal dynamics via the self-attention operation. The spatial temporal evolution of CO2 plume is predicted in parallel. Results show that the surrogate model can produce decent accuracy (R2 > 0.99) and offer a speedup of at least 80,000 times compared to numerical simulation. The proposed model is further integrated with the Particle Swarm Optimization (PSO) algorithm for the determination of well controls and perforation locations. The trapped CO2 volume and sweep efficiency are optimized. The proposed methodology significantly reduces the computational expenses of optimization tasks and provides a promising approach for GCS project management.