Comparison of Different Subjective Test Methods

Different test methods have been standardized for assessing video quality of 2D videos, e.g., ITU-T Rec. P.910 (Recommendation ITU-T P.910: subjective video quality assessment methods for multimedia applications, 2008) and ITU-R Rec. BT.500-13 (Recommendation ITU-R BT.500-13: methodology for the subjective assessment of the quality of television pictures, 2012). Due to the lack of a standardized method to assess the visual quality for 360 \({ }^\circ \) videos, studies have often used ITU-T Rec. P.910 (Recommendation ITU-T P.910: Subjective video quality assessment methods for multimedia applications, 2008) and ITU-R Rec. BT.500-13 (Recommendation ITU-R BT.500-13: methodology for the subjective assessment of the quality of television pictures, 2012) to evaluate the visual quality of 360 \({ }^\circ \) videos. ITU-T P.919 (Subjective test methodologies for 360° video on head-mounted displays. Recommendation P.919, 2020) was the first standardized recommendation on subjective test methods for 360 \({ }^\circ \) video – with contributions from the author. In this chapter, two different experiments are described and conducted in two different contexts in a lab setting. In the first experiment, the results of two different subjective test methods, M-ACR (Test 2) and DSIS (Test 3), were compared in two different contexts; namely, the M–ACR test was conducted in winter and the DSIS test in summer. In experiment 2, the results of three different subjective test methods, ACR (Test 4), M-ACR (Test 5), and DSIS (Test 6), were compared. These tests were conducted within a period of three weeks, thus having no difference in context in terms of seasons. Based on the reliability, test time, and general applicability of the procedure across different tests, a methodological framework for 360 \({ }^\circ \) video quality assessment is presented in this chapter.