This study develops a deep-learning pipeline for accurate three-dimensional (3D) reconstruction of the whole heart from sparse segmentations of wearable ultrasound (US) patch images. The research focuses on three key questions: the reconstruction accuracy using different arrays (linear vs. orthogonal), the impact of using one or multiple acoustic windows, and the effect of cardiac motion on reconstruction performance. We trained our models using simulated sparse volumes and dense ground truth volumes from 1000 pairs of CCTA images in best diastolic (BD) and systolic (BS) frames and tested them on another 1000 pairs. Results showed that the combination of two orthogonal arrays from apical and parasternal windows achieved the best reconstruction accuracy with a mean Dice Score of 0.95 for four cavities and left ventricle myocardium (LVM). The most cost-effective solution was using a single linear array in the parasternal long-axis (PLA) view, with a Dice Score of 0.87 for all seven labels, particularly effective for aortic reconstruction. Our motion simulation strategy demonstrated that even with significant cardiac motion, the reconstruction accuracy was well-preserved. This study provides valuable design recommendations and 3D shape reconstruction solutions for wearable US patches in cardiac monitoring.