The utilization of internet of things (IoT) devices in manufacturing industry is a promising trend which is changing our life while also brings about new challenges to smart factories, e.g., the invasion of illegal devices. These illegal devices may cause substantial damages to the smart factory by interfering the IoT systems or recording business secrets remotely. Moreover, it is hard to detect these illegal devices by conventional IoT security techniques or existing artificial intelligence (AI) based methods because of the lack of their priori information, owing to the fact that they are typically unknown. In this paper, we resort to radio frequency (RF) sensing techniques and propose a Long Short-Term Memory (LSTM) based method for illegal device detection in smart factory, such that the security can be effectively maintained for intelligent manufactory. In particular, the received RF signals are firstly analyzed, including both known signals from legal devices and those unknown signals from the illegal devices. In order to detect those illegal devices, we utilize the open set recognition approach without knowing their priori information in the training data, as well as utilize the LSTM neural network to catch the long-term dependence within the RF signals so as to increase the detection accuracy. It has been proved by the simulation results that the proposed method outperforms the existing AI based methods in detecting the unknown illegal devices.