A dynamic resource allocation problem for jointly optimizing access point selection and task offloading is proposed for a mobile edge computing system in an unmanned aerial vehicle (UAV)-assisted cell-free (CF) networks for the purpose of delay reduction in an urban scenario. On the one hand, given that various resources are coupled in the optimization problem, exhibiting non-convex traits and thus cannot be directly separated. On the other hand, to expedite algorithm convergence while ensuring user data security, we propose a method that combines the federated learning (FL) framework with deep reinforcement learning (DRL). Specifically, we introduce a federated deep Q network (F-DQN) algorithm to execute the dynamic resource allocation strategy for the proposed problem. Simulation results show that the proposed algorithm has better convergence performance than traditional Q-learning and deep Q network (DQN), and the proposed strategy has lower system delay with a maximum gain of 60.5% compared to other baseline strategies.