<p>On the Chinese space station, a space scientific experiment, named “The microgravity research on oscillation characteristics and transition issues of annular flow”, has completed its research plan for the first stage. The payload, supporting multiple annular fluid models with a heating, rotating or lifting central column, is developed for conducting a series of experiments in the fluid physics rack. This paper presents the space experiment with a fixed inner column and focuses on the volume ratio effect on the critical instability of thermocapillary convection. To observe the transition from steady flow to oscillatory flow, an infrared camera is adopted to record the temperature pattern of the annular flow. The instability critical point, critical frequency and instability mode are analyzed by the dynamic mode decomposition (DMD). The space experiment shows that the volume ratio(<i>Vr</i>) has a significant influence on the mode of thermocapillary instability by the surface configuration. The critical curves are divided into two branches according to critical frequencies, which are low-frequency modes (<i>f</i> = 0.056 ~ 0.072&#xa0;Hz, <i>Vr</i> = 0.625 ~ 1.065) and high-frequency modes (<i>f</i> = 0.301 ~ 0.419&#xa0;Hz, <i>Vr</i> = 0.525 ~ 0.575), respectively. The low-frequency modes include various azimuthal wave modes, i.e. regular wave (<i>m</i> = 3, <i>m</i> = 4) and irregular wave (<i>m</i> = 4 ~ 5). The high-frequency mode is the combination of radial waves and azimuthal waves. New coupled patterns, such as one spiral wave with <i>m</i> = 1 azimuthal mode, two spiral waves with <i>m</i> = 2 azimuthal mode and annular radial wave, etc., are discovered. Different modes are growing, coupling and competing during the onset of oscillatory instability. As volume ratio is changing, the mode transition is found as a continuous process of competition, rather than an abrupt change with clear volume ratio boundary.</p>