The proliferation of smart devices and the Internet of Things (IoT) has led to a substantial increase in the demand for multimedia services, necessitating high data rates in heterogeneous networks. This demand can be effectively addressed through the use of Unmanned Aerial Vehicles (UAVs), which are highly flexible and can be readily deployed to locations requiring network coverage. However, current wireless technologies rely on licensed spectrum. The scarcity of wireless spectrum makes it difficult to efficiently utilize frequency reuse without encountering co-channel interference. In this paper, we propose a UAV-based wireless network that delivers on-demand services using unlicensed spectrum. Using unlicensed spectrum also requires careful frequency reuse planning. We address the challenge of determining the optimal channel and transmission power for UAVs acting as aerial wireless access points. To tackle this issue, we present a deep reinforcement learning (DRL)-based solution that leverages tiny machine learning (TinyML) for on-device inference, taking into account the energy and computational constraints of the UAVs. Simulation results highlight the performance of our proposed solution when deployed on UAVs.