<p>Modern agriculture is reaping the benefits of automation by harvesting fruits and vegetables using robots that have adaptive grasping force depending on the compliance or softness of the fruit or vegetable. The successful operation of fruit picking using a robotic system with a gripper as an end effector hinges on its manipulation capability and ability to interact safely with the crop without affecting its physical properties. In this paper, we have proposed a novel hybrid gripper with six soft, passive auxetic structures attached to the rigid linkage, which has high shape conformability during the grasping of tomatoes through caging. The gripper is actuated through a scotch-yoke mechanism using a servo motor. A deep learning-based fruit localization algorithm has been developed to perform the picking operation of tomatoes through the gripper. A ROS-based framework has been used to control the robotic arm to achieve an optimized trajectory for fruit picking and to control the desired grasping forces along with the actuation of the pedicel cutter.</p>