Visual brain decoding is the process of reconstructing the images a person sees by analyzing their brain activity, as measured using functional MRI, electroencephalography, or magnetoencephalography. Recent advances in generative AI, particularly diffusion models, have significantly improved the semantic quality of the reconstructed images. In this work, we investigate the potential for improving the computational efficiency of brain-to-image models by focusing on a reduced number of informative embeddings. Specifically, we explore whether high-quality image reconstructions can be achieved by predicting a small, selected subset of CLIP-derived embeddings. We perform evaluations using the Natural Scenes Dataset, applying both a simple single-subject ridge regression model and a multi-subject federated learning framework adapted from a recent approach. The results demonstrate that embedding-efficient decoding can achieve competitive performance while substantially reducing model size, although with a decrease in quality on certain metrics. Code available at: https://github.com/IoanaGabor/efficient-vbd .