Traditional frameworks for autonomous flight of drones usually include environmental perception, map construction and path planning, which are rather cumbersome for deployment. Learning a neural network for flight trajectory prediction in an end-to-end manner can both avoid those complicated processes and improve the performance of trajectory planning, which directly maps the sensor data to the flight trajectory space. Therefore, we design a lightweight trajectory prediction neural network for fast flight in complex environments, in which an efficient transformer module is introduced to the decision layers for improving the expressiveness of network without increasing computationally complexity. In order to train and evaluate the trajectory planning network, we design a simulation platform based on UE4, AirSim, PX4 and EGO-Planner, which is easy to use for building experimental scenarios, generating training data, and evaluating trajectory planning network models. Extensive experiments are conducted in the developed simulator to evaluate the performance of the proposed prediction network. The results demonstrate the effectiveness of our methodology.