Universal Handwriting Recognition for Mobile Devices via Acoustic Sensing

Handwriting input is essential for efficient human-computer interaction on small-screen mobile devices, as it provides a more intuitive and convenient method compared to soft keyboard typing. One promising approach for handwriting recognition is based on passive acoustic sensing, which offers advantages such as low cost, universal availability, and robustness to various factors. However, its performance on cross-user problem still needs to be further improved, especially for free-style lowercase input. In this paper, we propose TransWriter, an acoustic-based handwriting recognition system suitable for mobile devices. It utilizes the built-in microphone to collect acoustic signals generated by the pen scratch on the surface. We perform data augmentation based on the key factor of writing speed to enhance the diversity of the training dataset. Furthermore, we propose a deep network architecture combining LSTM and transformer models, which can capture local features and global features, retain more spatial information, and models the temporal relationship between time frames. To validate the effectiveness of TransWriter, we implement a prototype system on a mobile phone and conduct an extensive experimental evaluation. The results of our experiments demonstrate that TransWriter achieves an accuracy of 80.77% for unseen users, across various practical scenarios.