We present FQ-MinMax, a federated Q-learning framework ... finite state and action spaces. We establish tight lower bounds showing that achieving any collaborative speedup in sample complexity necessitates at least \( \varOmega \!\left( \frac{1}{1-\gamma }\log N\right) \text { communication rounds} \quad \text {and}\quad \varOmega \!\left( \frac{|S||A|}{1-\gamma }\log N\right) \text { bits per agent}, \) where \(\gamma \) is the discount factor and N denotes local sample size. FQ-MinMax meets these limits by combining minibatched updates, variance reduction, and quantized message passing. It attains a sample complexity of \( \widetilde{O}\!\left( \frac{|S||A|}{M\varepsilon ^2(1-\gamma )^3}\right) \text { per agent,} \quad \widetilde{O}\!\left( \frac{1}{1-\gamma }\right) \text { rounds,} \quad \widetilde{O}\!\left( \frac{|S||A|}{1-\gamma }\right) \text { bits.} \) Extensive simulations validate the algorithm’s performance, demonstrating significant gains in scalability and efficiency with minimal communication overhead.