Federated matrix factorization (FedMF) has recently emerged as a privacy-friendly paradigm which runs matrix factorization (MF) in a federated learning (FL) setting and enables users to keep their individual rating data local in the training process. In FedMF, users only need to share out item gradients for aggregation. However, prior work has shown that the item gradients, if directly exposed, can still leak users’ rating data. Meanwhile, the rating data are typically sparse and simply uploading the gradients of rated items will leak which and how many items a user rates. In light of the above, in this paper, we present SARA, a new sparsity-aware efficient oblivious aggregation service for FedMF. SARA protects the confidentiality of item gradients as well as hides which and how many items a user rates, through a custom sparsity-aware design that delicately builds on differential privacy and lightweight cryptography. Extensive experiments over real-world datasets demonstrate that SARA is utility-preserving and can bring the users significant savings (up to 98.28%) in the number of transmitted item gradients, compared to the baseline of fully transmitting gradients for all possible items.