<p>This paper introduces a hybrid scrambling method that uses both pixel-level and block-level scrambling, along with a chaotic diffusion process based on a novel spiral shell 3D hyperchaotic system. The novel spiral shell 3D chaotic system was studied using Lyapunov exponents, bifurcation diagrams, and multistability, and it was tested successfully with the NIST test. The chaotic system was also successfully implemented in the ESP8266, confirming its feasibility for real-time applications. The proposed encryption algorithm uniquely processes all three RGB channels simultaneously, maintaining critical inter-channel correlations while performing pixel-level scrambling using the kernel matrix. Followed by iteratively dividing the image into 4 × 4 subblocks and then scrambling using the Inter Woven Kolam (IWK) matrix. Further, the scrambled image was XORed with the random sequence generated by the spiral shell 3D chaotic system. The encryption method resists statistical and differential attacks. The key space is optimized at 2⁵⁸⁸, providing significant security against brute-force attacks. The scheme also has high entropy (7.99) and low correlation coefficients (− 0.0022, 0.01193, and 0.00616) between neighboring pixels. The differential attack study shows near-optimal results, with a UACI of 34.65% and an NPCR of 99.62%, indicating good input sensitivity and resistance to differential attacks. Numerous experiments using simulated noise and crop attacks on cipher images evaluate the algorithm’s resistance. The proposed method has achieved high resistance for a single round of iteration, and the experimental findings show that it is appropriate for secure data transfer.</p>