Optimizing the process of flow boiling and improving heat transfer efficiency require preventing the onset of flow instability (OFI) during the operation. The aim of this work was to examine the instability of upward flow in a rectangular mini-channel with a cross-section of 2 \(\times \) 2 mm when the heat flux and mass flux were gradually increased. The study examined the periodic transition of flow patterns during OFI, calculated the impact of bubble coalescence on OFI using the growth rate of bubbles with the potential to form gas columns, and introduced entropy generation to assess the irreversibility and disorder of the system. The findings revealed that the formation of gas columns during OFI is the main cause of the periodic transition of flow patterns. When the inlet subcooling \({\Delta T}_{\text{sub}}\) is 12.1 \(\text{K}\) , the average number of bubble coalescences during OFI is 240% higher than during stable flow. Pressure drop fluctuations are significantly impacted by bubble coalescence, which also contributes to the formation of gas columns and an increase in system instability. The pressure drop and the heat transfer coefficient are inversely related. Lowering the inlet subcooling and reducing mass flux are like to cause OFI. Entropy generation analysis indicates that reducing inlet subcooling and increasing fluid velocity can reduce the system's irreversibility. When OFI occurs, entropy generation rises sharply.