<p>Sr₂IrO₄ is one of the prime candidates for realizing exotic quantum spin orders owing to the subtle combination of spin-orbit coupling and electron correlation. Sensitive local magnetization measurement can serve as a powerful tool to study these kinds of systems with multiple competing spin orders since the comprehensive study of the spatially-varying magnetic responses provides crucial information on their energetics. Here, using sensitive magneto-optical Kerr effect measurements and spin Hamiltonian model calculations, we show that Sr₂IrO₄ has non-trivial domain structures which cannot be understood by conventional antiferromagnetism. This unconventional magnetic response exhibits broken symmetry along the Ir-O-Ir bond direction and is enhanced upon spin-flip transition or Rh-doping. Our analysis, based on possible stacking patterns of spins, shows that introduction of an additional rotational-symmetry breaking is essential to describe the magnetic behavior of Sr₂Ir_1-xRh_xO₄, providing strong evidence for a nematic hidden order phase in this highly correlated spin-orbit Mott insulator.</p>