<p>Stochastic optimization algorithms have become the majority of global optimization algorithms. In order to evaluate a stochastic optimization algorithm, it is necessary to access the random perturbation around its average performance. This paper considered the average performance of a stochastic optimization algorithm, i.e., the performance of the average values of solutions found in dozens of independent runs for given computational cost. Two popular average performances namely the the mean performance and the median performance are considered. Firstly, the mean performance is shown numerically to be possible worse than each independent run in set-based data analysis methods including the popular data profile method and some other similar methods. The underlying reason lies in a specific data structure: the performance matrix exhibits sparsely distributed outliers that are uniquely associated with different runs for each problem. We have shown with theoretical and numerical analysis that the condition “<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(p+q&gt;1\)</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>p</mi> <mo>+</mo> <mi>q</mi> <mo>&gt;</mo> <mn>1</mn> </mrow> </math></EquationSource> </InlineEquation>” is needed to guarantee the average performance to work well. This condition requires that the average performance can “solve” enough problems and sufficient solutions are worse than the average value. Under this view, we have shown that the mean performance is often not suitable in the data profile method, and the median performance is often much better. Extended numerical results have supported our theoretical and numerical analysis.</p>