Galaxy evolution is an area of vital importance in current research as it is believed to hold vital clues of the past as well as the future of the universe. The structure or morphology of a galaxy acts as an indicator of its stage of evolution and may also shed light on the course of its future evolution. With the availability of large amounts of telescopic image data, such as from the recent James Webb telescope, the deployment of high-performance machine learning and deep learning techniques has been largely facilitated. Distinguishing between the various types of Spiral and Elliptical galaxies is the primary objective that needs to be fulfilled. Existing works have only limited application potential due to their heavy-weight structure. On the other hand, transformer-based models achieve high accuracy but lightweight models usually fail to produce an acceptable accuracy. The uniqueness of the proposed work lies in its approach of relying on cascaded simple Zoom-Dilate Convolution blocks to achieve most of the classification accuracy while transformer blocks are used to raise it further but not going overboard with it to keep the model reasonably lightweight. Another major contribution of this study is that it not only clearly establishes how a judicious balance between accuracy and the depth of the network is achieved but also indicates how this may be improved further.