This work aims at the drone-view building localization of matching the drone-view video with the satellite-view image. As one of the primary tasks, drone-view building localization could facilitate many potential applications such as precise drone delivery. Yet there is no public drone-view video dataset. Considering the limitation, we introduce a new large-scale synthetic dataset for video-based building localization, named Building-632, which contains videos of 632 buildings from both drone and satellite platforms. Compared with image-based datasets, Building-632 is featured in three aspects. First, Building-632 mimics the real-world drone flight via dense flight simulation over two towns, resulting in 4572 videos. Second, the proposed dataset is large-scale, in that it consists of over 2200 annotated bounding boxes with sequential information from over seven viewpoints for each building. Third, Building-632 also provides rich flight metadata, including GPS coordinates and the flight path. Extensive evaluation of the state-of-the-art methods, including prevalent CNN and LSTM, is presented. The experiment provides several insights into developing a robust building localization system, and we hope all these efforts could ease the future study for this research field.