Convolutional neural networks (CNNs) have demonstrated exceptional capabilities across diverse visual recognition tasks. While network pruning serves as a vital technique for compressing overparameterized models effectively, recent investigations reveal its unintended consequences in exacerbating bias among demographic subgroups. Although some research has identified potential factors contributing to unfairness, they lack practical guidance for mitigating it. Driven by the proven efficacy of geometric factors of features (i.e., length and angle) in enhancing robustness of pruned model, we systematically explores their underexplored connections with fairness in pruned networks through comprehensive empirical evaluations. Building upon these insights, we develop FairGEO (Fairness in pruning based on GEOmetric factors of features), a lightweight framework that adjusts the disparities in the geometric factors across subgroups to improve fairness in pruning. Evaluations on multiple image classification benchmarks demonstrate FairGEO can improve fairness while maintaining accuracy compared to pruned baseline and its integration into existing pruning methods confirms its generalizability in enhancing fairness in pruning.