In an online environment, traditional security mechanisms such as operational and encryption methods are not effective against social engineering attacks. These attacks deceive users regardless of the security mechanisms used. Therefore, several techniques based on machine learning have been developed to detect social malicious activity in an online environment. However, most of these techniques require scarce labeled data, which is difficult to obtain, and present the inability to predict new malicious activity. Thus, in this paper, we present an anomaly-based technique that uses machine learning algorithms over continuous data stream for detecting social engineering attacks in an online environment. Our approach is stream-based and unsupervised to overcome the main challenges that happen when we use a batch supervised machine learning approach. We analyzed the Half-Space Trees and One-Class SVM algorithms that we have adapted to stream processing to verify which one has the best effectiveness and efficiency in an online environment. The analysis of combinations of the algorithms with feature extraction schemes from textual data shows that the models can be used efficiently and effectively for anomaly detection to replace the binary classification task.