Automatic-releasing tools are widely used in neurosurgical procedures to drill holes in the skull while protecting the brain from drilling through. The application of these tools in robotic surgical systems faces challenges due to the difficulty in understanding the skills of experienced surgeons, their complexity, the curved surfaces of the skull, and the need for consistent force control in variable bone layers. By applying machining theory and material removal analysis with real-time surgical force data, the drilling through detection method of bone layers is established based on the thrust force distribution model on cutting edges. Further, the effects of multi-layer and surface misalignment are discussed to explain the determination of the force threshold used in recognition, which translates the surgeon’s skills into the control design principle. The robotic-assisted drilling controller is then designed with a switching guidance force based on the admittance controller, which reduces operational burden and enhances surgical efficiency. The experiments of drilling on complex surfaces and bone layers address the consistency, stability, precision, and rapid response of specialized drill bits in skull layers, which results in a safety improvement in robotic-assisted craniotomy.