This study presents an exploration into the application of advanced machine learning attention-based model Temporal Fusion Transformer for modelling streamflow in peninsular India. Leveraging the abundance of historical meteorological and hydrological data, the study examines the utility of machine learning models for streamflow modelling using multiple strategies. Through comprehensive evaluations, the study analyzes the efficacy of the Mixture of Expert models under different conditions. The findings provide valuable insights into the potential of extensible machine learning models for streamflow modelling, shedding light on their applicability in hydrological studies. The results contribute to advancing our understanding of using large-scale machine learning models for analyzing streamflow.