<p>Stability lobe diagram is essential to determine chatter-free machining parameters for process planning. A tough issue in predicting milling stability is that the crucial inputs tool tip dynamics vary according to tool-holder combination conditions. It is quite time-consuming to update tool tip dynamics through repeated experiments or simulations. To address this issue, this paper proposes a transfer learning and optimization technique-based inverse stability solution to identify the overhang length-dependent tool tip dynamics for different tool-holder combinations. Impact tests are carried out under multiple overhang lengths of a source tool and limited overhang lengths of a target tool. Obtained tool tip FRFs are used to generate analytical stability limits for constructing the source and target datasets respectively. They are combined to iteratively train a regression model for predicting target overhang length-dependent stability limits using instance-based transfer learning, where the domain adaption is implemented by the multi-fidelity surrogate model. Target stability limits under each overhang length are taken to construct an optimization model whose variables are the modal parameters, which is solved by an improved teaching–learning-based optimization algorithm. A detailed experimental study demonstrates that this proposed method can efficiently and accurately predict overhang length-dependent modal parameters requiring fewer impact tests.</p>