Due to the lack of compact and accurate models for computing the aerodynamic moment on ducted fan, existing model-based control approaches hardly achieve satisfactory performance on the attitude control of ducted fan unmanned aerial vehicles (UAVs), especially in the presence of disturbance. Alternatively, we can transition towards a sensor-based approach, namely incremental nonlinear dynamic inversion (INDI), which requires less modeling of the system dynamics. In this paper, we propose an attitude controller for a small ducted fan UAV using INDI. In the INDI design, the system nonlinearities and the effects from disturbance could be well compensated via high-frequency sensor data sampling. Furthermore, this study demonstrates how INDI provides an essential interface for control allocation in ducted fan UAVs, where actuators are typically redundantly configured, leading to significant control allocation challenges. In the control allocation design, we propose a prioritized control allocation (PCA) algorithm by which the linearization from INDI is to be attained in priority, aiming to sustain the stability of the closed-loop system. Finally, the proposed INDI+PCA approach is validated through several comparative experiments, achieving satisfactory results in both reference tracking and disturbance rejecting.