Quantum computing represents a central challenge in modern science. Neutral atoms in optical lattices have emerged as a leading computing platform, with collisional gates offering a stable mechanism for quantum logic1–10. However, previous experiments have treated ultracold collisions as a dynamically fine-tuned process11–22, which obscures the underlying quantum geometry and quantum statistics crucial for realizing intrinsically robust operations. Here we propose and experimentally demonstrate a purely geometric two-qubit SWAP gate by transiently populating qubit doublon states of fermionic atoms in a dynamical optical lattice. The presence of these doublon states, together with fermionic exchange anti-symmetry, enables a two-particle quantum holonomy—a geometric evolution in which dynamical phases are absent23. This yields a gate mechanism that is intrinsically protected against fluctuations and inhomogeneities of the confining potentials. The resilience of the gate is further reinforced by time-reversal and chiral symmetries of the Hamiltonian. We experimentally validate this exceptional protection, achieving a loss-corrected amplitude fidelity of 99.91(7)% measured across the entire system consisting of more than 17,000 atom pairs. When combined with recently developed topological pumping methods for atom transport16, our results pave the way for large-scale, highly connected quantum processors. This work introduces a new model for quantum logic that transforms fundamental symmetries, including quantum statistics, into a powerful resource for fault-tolerant computation.