Wildfire is a kind of frequent disaster that often harms human lives and properties. To implement early warning of wildfires, a real-time smoke detection method using convolutional neural network is proposed in this paper. Within the network, a lightweight CSP-Darknet backbone combined with the feature pyramid grids is designed for multi-scale feature extraction. Accordingly, we introduce a feature enhancement module to fuse the spatial information of smoke for improving the representation ability. Moreover, we design a motion aware module to capture temporal features of smoke from the input consecutive dual-frames. The motion aware module could generate an attention mask, which can be multiplied by the output from feature enhancement module so as to merge the spatial and temporal smoke characteristics. The experimental results on the publicly available datasets FIgLib and Smoke100K validate the effective performance of the proposed method for smoke detection.