Punctuation and word casing prediction are crucial components of automatic speech recognition (ASR) systems. With the popularity of on-device end-to-end streaming ASR systems, the on-device punctuation and word casing prediction become a necessity while we found little discussion on this. With the emergence of Transformer, Transformer-based models have been explored for this scenario. However, Transformer-based models are often too large to be practical for on-device ASR systems. In this paper, we propose a lightweight and efficient model that jointly predicts punctuation and word casing in real time. Our model combines the strengths of convolutional neural networks (CNN) and bidirectional long short-term memory (BiLSTM) networks. Experimental results on the IWSLT2011 test set demonstrate that our proposed model achieves a 9% relative improvement in overall F1-score compared to the best non-Transformer models. Furthermore, our model achieves comparable performance to a representative Transformer-based model while being only one-fortieth of its size and 2.5 times faster in inference speed. This makes it particularly suitable for on-device streaming ASR applications. Our code is publicly accessible for further research and development.