Few-shot dense retrieval (DR) aims to generalize effectively to novel query scenarios by learning from a few samples. Despite its importance, there has been limited study on the few-shot learning abilities of DR models. One primary reason for this gap is the absence of standardized datasets and evaluation protocols. As a result, current methods often resort to random sampling from regular datasets to create “few-data” setups, making robust and reliable evaluations challenging. This paper introduces a customized FewDR dataset and evaluation benchmark for few-shot DR. Specifically, FewDR categorizes query-answer entity pairs based on their entity relations into classes and organizes queries and relevant passages containing both entities into shots. We then use class-wise shot sampling on FewDR to establish a standardized “few-shot” setup. This evaluation setting reduces variability across different sampling trials, enhancing the robustness of evaluating DR’s few-shot performance. Our extensive empirical findings across 17 DR models reveal the considerable challenges these models face in standard few-shot learning scenarios. Our code and data are publicly available at https://github.com/SunSiShining/FewDR .