Securing sensitive visual data is essential in the digital age. Our research presents a sophisticated model of image encryption that combines RNA pixel conversion with an autoencoder. A convolutional neural network (CNN) is used for efficient feature extraction. Convolutional encoding and RNA pixel conversion, which map pixels to RNA sequences (Adenine, Cytosine, Guanine, Uracil) enabling stronger encryption, are the model’s primary innovations. By guaranteeing that every encryption instance is distinct, this dynamic technique strengthens security. Our approach ensures security and picture integrity by demonstrating robust encryption capabilities with minimum reconstruction loss (0.0617), as tested on the MNIST dataset. Furthermore, the encryption procedure is optimized by a fractional order optimizer, FGL_Adam, which incorporates weight decay, learning rate modifications, and a security improvement factor.