Intra-Voxel Incoherent Motion (IVIM) is a form of diffusion MRI that seeks to characterize the motion of microvascular blood under the assumption that this motion of blood is a pseudo-diffusion process that acts like a random walk. IVIM MRI has been applied to estimating tissue perfusion and hemodynamics in the brain and has potential for inferring microvascular architecture. Here we employ biophysical simulations based on the Vascular Anatomical Network (VAN) modeling approach, which provides both realistic microvascular anatomy and dynamics, to simulate IVIM data at a single location in the cerebral cortex. We demonstrate that the microvascular anatomy and physiology yield asymmetries in the pseudo-diffusion of blood and an IVIM signal behavior that is consistent with multiple blood pools moving with different velocities. We also provide estimates of IVIM signal decay with increasing diffusion weighting for both mouse and human cerebral cortex. This modeling framework can help interpret modern IVIM MRI data and help guide new experiments to extract subvoxel anatomical and physiological information from motion-encoded MRI.