Visual servoing is a well-established technique for object grasping and controls the robot in a closed-loop fashion. It typically uses hand-crafted features or a neural network that directly learns the control output. We propose an alternative approach that uses an off-the-shelf neural network object classifier and can therefore compute target poses without manually selected features while also not requiring training from scratch. Instead, the object classifier only needs to be fine-tuned on a domain-specific dataset, significantly reducing the amount of required training data. We describe a task sequencing approach that can control a robot with a mobile base and an additional gripper, which is a typical setup in many robotics applications. We evaluate the approach in the RoboCup Logistics League and demonstrate the reliability and speed of the proposed approach.