This research presents an innovative method to enhance the effectiveness of a Convolutional Neural Network (CNN) classifier when dealing with offline handwritten text. By adapting to the writing style of individual users, this method achieves lightweight and privacy-preserving classification without requiring specialized deep-learning frameworks or hardware on the user’s end. The training phase, which takes place on the server side, involves training the base CNN and defining the writing styles of different characters through clustering in the feature space learned by the CNN. In the adjustment phase, implemented on the user side, the system determines the user’s writing style based on their writing and corrections of recognitions offered by the base CNN. This information is integrated into an ensemble of alternative k-NN classifiers with a lightweight Bayesian selection mechanism that learns when to correct the base CNN. Essentially, this ensemble construction mechanism conducts automated error-space analysis of the base CNN to improve its decision-making. To evaluate the proposed method, we tested it on two widely-used datasets - NIST Special Database 19 and ETH Zurich Deepwriting dataset - achieving up to 2.7% improvement in classification accuracy. This resulted in state-of-the-art classification results on these datasets.