The construction of strong encryption techniques is crucial to meet the increasing demand for secure transmission as well as storage of medical images. A substitution box (S-Box) is an important component of block ciphers and nonlinearity is an important attribute to consider while designing secure S-boxes. As a result, it is required to create new approaches for producing S-boxes with high non-linearity scores. We present a method of parametrization of the generalized triangle group \( \left\langle x,y\kern1em |{x}^2={y}^5={w}^k=1\right\rangle \) as linear groups, where \( w= xyx{y}^2x{y}^4 \) which is extended by the parametrization for triangle group \( \left\langle x,y,t\kern1em |{x}^2={y}^5={t}^2={(xt)}^2={(yt)}^2={(xy)}^k=1\right\rangle \) . This parametrization is then used for the construction of a highly nonlinear and secure substitution box designed for \( {2}^8 \) elements, tailored specifically for the finite generalized triangle group case with \( k=2 \) for \( \theta =64 \) which is parameter for all homomorphism from \( {H}_5 \) to \( PSL\left(2,q\right) \) , possessing an order of 1200. We rigorously evaluate and analyze various common security indicators associated with the proposed substitution box. The proposed S-box is evaluated for picture encryption using various statistical approaches. Comparative analysis and additional scrutiny reveal promising attributes, affirming its suitability, efficacy, and potential applicability in the domain of medical image encryption. Our S-box achieves the necessary conditions for secure communication as well as image encryption, as confirmed by positive examination results.
