Multi-agent trajectory data collected from domains such as team sports often suffer from missing values due to various factors. While many imputation methods have been proposed for spatiotemporal data, they are ill-suited for multi-agent sports, where player movements are highly dynamic and interactions evolve over time. To address these challenges, we propose MIDAS (Multi-agent Imputer with Derivative-Accumulating Self-ensemble), a data-efficient framework that imputes multi-agent trajectories with high accuracy and physical plausibility. It jointly predicts positions, velocities, and accelerations via a Set Transformer-based neural network and refines them by recursively accumulating predicted velocity and acceleration values. These predictions are then combined using a learnable weighted ensemble to produce final imputed trajectories. Experiments on three sports datasets show that MIDAS significantly outperforms existing baselines, with particularly large margins in limited-data settings. We also demonstrate its utility in downstream tasks such as estimating total distance and pass success probability. The source code is available at https://github.com/gkswns95/midas.git .