Fabric classification from microscopic imagery poses unique challenges due to subtle inter-class variations and the absence of large-scale annotated datasets. In this work, we present a comprehensive study on the effectiveness of handcrafted optical descriptors—namely Gabor filters, Haralick textures, Local Binary Patterns (LBP), and luminance—in distinguishing fabric types. Diverging from deep learning paradigms, we adopt a structured pipeline leveraging XGBoost, selected through progressive experimentation with KNN, Random Forests, and LightGBM. To assess robustness, we introduce perceptual shifts via controlled lighting transformations, where the golden hour condition consistently improved model fidelity. Our method attains 71.43% accuracy across seven fabric categories, rising to 83.5% upon excluding visually ambiguous classes. These results demonstrate that classical features, when coupled with ensemble learning and perceptual augmentation, offer a strong and interpretable alternative for fabric recognition in constrained visual domains.