Pupil segmentation extracts the pupil area of human eyes to track the eye movement. It is the first step for most of the vision-based gaze tracking systems which have attracted significant interests from both academic and industrial communities. When incorporated in commercial VR/AR devices, the accuracy, robustness and efficiency of the pupil segmentation is the fundamental for the successful usage of gaze tracking in human computer interaction, foveated rendering and etc. In this paper, we propose, KD-Eye, a lightweight vision-based pupil segmentation approach to realize accurate and efficient estimation of pupil regions. We introduce a coarse-to-fine strategy to significantly reduce the computation of the algorithm and complexity of the network. Then knowledge distillation is applied to guide the training of a lightweight student network with a large teacher network trained by a public dataset. According to our evaluation on real-world dataset and implementation on a VR platform, the coarse-to-fine strategy and lightweight network can speed up the segmentation process by over 240 times and the knowledge distillation can improve the accuracy of the student network especially when the number of training samples are limited.