Neural rendering has shown promising potential for 3D reconstructing and visualizing ultrasound data; however, long training times pose a critical barrier for clinical adoption. In this work, we propose a novel two-step grid-based approach to address this efficiency gap. First, we build a rough volume by aggregating ultrasound frames that generates an approximation of the scanned region. Second, we refine this volume by learning local, view-dependent spherical harmonics (SH) coefficients on a per-voxel basis. By initializing the zero-order SH term from the aggregated volume and updating coefficients only in spatially relevant neighborhoods, the method dramatically reduces training overhead compared to other popular approaches. Across four ultrasound datasets (including simulations and real phantom recordings), our approach achieves a speedup of approximately 3 \(\times \) in reaching a strong baseline performance level while providing comparable or superior reconstructions at early training stages. These results highlight the feasibility of fast neural rendering for ultrasound and open new avenues for real-time 3D reconstructions in medical imaging.