Application Identification Using Pairwise Classification

Cluster computing platforms are being increasingly used for processing big data applications. Cluster operators use historical data collected from executions of these for purposes of scheduling, resource allocation, and execution time target estimation. When referring to historical data to retrieve corresponding records related to an application, the application names are often used as a label. However, cluster users can assign custom names to their applications each time they submit them for processing, and different users may use different naming conventions for the same application they may submit. This creates challenges for finding matching historical data. In this paper, we propose a method that identifies and labels applications based on their execution logs. This can allow cluster operators to identify and label the same applications without relying on application names. Our method relies on pairwise classification, which does not require extensive training data. We have evaluated our proposed pairwise classification technique with four different learning algorithms and 13 big data applications. Our method works for new application types that were not observed during the model training with high accuracy. Our best-performing algorithm achieves 92% accuracy.