We present a sharp interface immersed boundary method in a block-structured adaptive grid framework to investigate the dynamic interaction of rigid bodies in three-dimensional, incompressible viscous fluid flow. We employ a higher-order level set representation of the immersed interface while utilizing an Eulerian treatment of the fluid. We enforce no-slip and no-penetration boundary conditions using a ghost cell approach, achieved through a fourth-order accurate least-squares reconstruction of flow variables within these ghost cells. The governing flow equations are discretized on staggered adaptive grids, utilizing central schemes, and the resulting discrete equations are numerically solved using an exact projection method. The projection on adaptive grids with a level set description of the interface makes the method robust and computationally efficient for three-dimensional flows involving stationary or moving rigid bodies. Various test cases are presented, like flow past a stationary sphere, a translating sphere in an initially quiescent fluid.