With the advancement of marine fisheries, detecting abnormal fishing vessel behavior is crucial for marine resource management and the analysis of ship encounter scenarios. However, traditional methods based on static rules or supervised learning struggle to adapt to the dynamic operations and complex trajectory characteristics of fishing vessels, often resulting in high false alarm rates and insufficient interpretability. To address this issue, this paper proposes an analytical framework integrating spatiotemporal trajectory features and multi-stage clustering for detecting abnormal fishing behaviors in ship encounter scenarios. First, a dual-filtering mechanism is designed to preprocess the original AIS trajectory data. Then, a hybrid spatiotemporal encoder with MiniRocket is constructed, utilizing a dual-path LSTM-Transformer architecture to capture both local motion patterns and long-range dependencies in trajectories. To better represent non-stationary movements, randomized convolutional kernels are introduced. Additionally, VQ-EMA technology dynamically compresses high-dimensional features. Next, a multi-stage density clustering optimization strategy is proposed to precisely extract complex trajectory patterns, with fragmented clusters merged using Hausdorff distance metrics. Finally, an anomaly indicator system is established based on speed standard deviation, heading angle variance, and stop event frequency. The framework accurately distinguishes fishing vessel states (e.g., trawling, gillnetting) and drifting hazard. Practical validation confirms its unsupervised anomaly detection capability. It fully aligns with COLREGs regulations.