This study examines the performance of pre-trained Vision Transformers (ViTs) with architectural modifications for identifying the four Khasi dialects, namely Sohra, Mairang, Maram, and Nongkrem. The Multi-Layer Perceptron (MLP) hidden dimension, activation function, and classification head were modified to analyze their effects. Modifying the MLP size, either by decreasing or increasing it, results in lower performance compared to the original MLP size. It was observed that using various activation functions such as Rectified Linear Unit (ReLU), Sigmoid Linear Unit (SiLU), and Leaky Rectified Linear Unit (Leaky ReLU) of the MLP block did not improve the performance. Further, it was found that modifying the classification head by adding more layers reduced the performance accuracy of the model. These findings suggest that the modifications made to the pre-trained ViT model do not enhance its performance on the Khasi dialect identification task.