Stroke classification remains challenging due to variations in writing style, ambiguous content, and dynamic writing positions. The core challenge in stroke classification is modeling the semantic relationships between strokes. Our observations indicate that stroke interactions are typically localized, making it difficult for existing deep learning methods to capture such fine-grained relationships. Although viewing strokes from a point-level perspective can address this issue, it introduces redundancy. However, by selecting reference points and using their sequential order to represent strokes in a fine-grained manner, this problem can be effectively solved. This insight inspired StrokeNet, a novel network architecture encoding strokes as reference pair representations (points + feature vectors), where reference points enable spatial queries and features mediate interaction modeling. Specifically, we dynamically select reference points for each stroke and sequence them, employing an Inline Sequence Attention (ISA) module to construct contextual features. To capture spatial feature interactions, we devised a Cross-Ellipse Query (CEQ) mechanism that clusters reference points and extracts features across varying spatial scales. Finally, a joint optimization framework simultaneously predicts stroke categories via reference points regression and adjacent stroke semantic transition modeling through an Auxiliary Branch (Aux-Branch). Experimental results show that our method achieves state-of-the-art performance on multiple public online handwritten datasets. Notably, on the CASIA-onDo dataset, the accuracy improves from 93.81 \(\%\) to 95.54 \(\%\) , demonstrating the effectiveness and robustness of our approach.