<p>The robustness of an algebraic-based encryption system relies on its fundamental algebraic operations. The complex algebraic structure provides a significant advantage in ensuring security against powerful cryptanalysis. The purpose is to establish substitution boxes with complex algebraic structures based on multivariable polynomials, as typical S-boxes rely on algebraic structures based on univariate polynomials. The study proposes the use of non-chain rings with bivariate polynomials to construct S-boxes, which increases the algorithm’s algebraic complexity and enhances the efficiency of image data security. We constructed two S-boxes generated from the subgroup of the group of units of the ring. A subset of a group is formed by an S-box that fulfills all axioms of the group structure, except the closure axiom. The lookup table is constructed using a set that does not satisfy group axioms but contains the inverse of each element, addressing another challenge. The other subgroup is mapped to the Galois field. The S-box of the ring is used for the substitution process and the other S-box is used for the diffusion process. Next, the key is used for the exclusive OR operation with the image. Finally, the encrypted image experiences different types of differentials, statistical, algebraic, and NIST attacks, demonstrating the efficacy of the proposed scheme.</p>