This paper explores the application of Convolutional Long Short-Term Memory Networks (ConvLSTM) to extract shape parameters from the fins of heat exchangers and subsequently employs reinforcement learning to optimize their design. Heat exchanger fins play a crucial role in enhancing thermal performance by increasing the surface area available for heat transfer. However, the optimization of fin shapes often involves complex geometrical and thermal considerations. By utilizing ConvLSTM, we can efficiently capture intricate features and temporal variations of fin profiles from sequences of images, transforming these shapes into quantifiable parameters. These parameters serve as inputs for a reinforcement learning model that iteratively improves fin designs based on their thermal performance metrics. This approach not only simplifies the design process but also enables the exploration of a wider design space, ultimately leading to more efficient heat exchanger configurations. Experimental results demonstrate the effectiveness of ConvLSTM in accurately extracting relevant features from time-series data and the capability of reinforcement learning in optimizing heat exchanger fin designs. Our findings contribute to the development of smarter heat exchanger systems with enhanced performance, showcasing the potential of combining deep learning techniques with traditional engineering practices.