The problem of optimal formation-containment control for groups of heterogenous unmanned air-ground vehicles (UA-GVs) under switching topologies is solved using reinforcement learning (RL) techniques. The quadrotor dynamics exhibit underactuation, and the dynamics of the UA-GV system are nonlinear and involving uncertain parameters. Positional estimators are devised for each agent to deliver references subject to the impact resulted from topological changes. Optimal control strategies are formulated without precise knowledge of the inertial parameters of the agents. Simulation examples are demonstrated, confirming the efficacy of the devised optimal control strategies.