This paper proposes a novel approach to handwriting similarity detection using a lightweight Patch-Based Multi-headed attention Similarity Network (PatchSimNet). Unlike traditional text-dependent approaches, PatchSimNet tackles the more flexible text-independent binary classification tasks by determining whether two handwriting samples were produced by the same individual, regardless of the text content. The model processes handwriting images by dividing them into patches and applying a multi-headed attention mechanism, enabling the model to effectively analyse the nuanced features such as stroke style and spatial alignment. Notably, PatchSimNet requires less data and eliminates the need for preprocessing steps such as word segmentation, simplifying the analysis pipeline. These extracted features are compared to assess the similarity between samples, offering a robust approach to handwriting analysis. With a lightweight architecture (~220.52 MB), the model ensures computational efficiency, making it an ideal candidate for resource-constrained scenarios such as mobile devices or real-time forensic investigations. The model was evaluated on the IAM dataset, achieving a peak classification accuracy of 95.82%. The classification report reflects, among the two classes: Precision and F1 Score of 96%, Recall of 96% and 95% for same and different writer respectively. With its exceptional results, PatchSimNet bridges the gap between traditional forensic methods and modern deep learning, paving the way for innovative applications in document verification, fraud detection, and legal investigations.