Tall Array Method Efficiency in Computing Statistics of Big Data
摘要
A problem of determining real efficiency of the Tall Array method in computing statistics of numeric data is considered. Random arrays of specified numeric types are generated, for which computational time is measured by applying ordinary and Tall Array MATLAB operators in computing statistics of the generated arrays. The main finding is, for two parallel processor workers, the Tall Array method becomes efficient if an array of no fewer than a 10 to 35 million entries is to be processed. The Tall Array minimum and maximum are similarly efficient in processing arrays with double and single precision, both real-valued and complex-valued, and their efficiency is comparable to that of the Tall Array standard deviation. The efficiency threshold of the Tall Array covariance is 1.75 to 5 times less than that of the Tall Array standard deviation. The Tall Array mean is the least efficient for non-integer representation. The Tall Array operators of minimum, maximum, and mean are similarly efficient on unsigned and signed integers. Nevertheless, the Tall Array mean operator appears to be more efficient on signed integers. Moreover, as the number of bits is increased, its efficiency grows, where the efficiency threshold decreases from 20 million entries for int8 down to 5 million entries for int64.