Let G be a connected, simply connected three-dimensional Lie group (unimodular or non-unimodular) equipped with a left-invariant (Riemannian or Lorentzian) metric g. By definition, the isometry group \(\textrm{Isom}(G, g)\) contains G itself, acting by left translations. It turns out that, generically, \(\textrm{Isom}(G, g)\) is actually equal to G, and the natural question then becomes to classify those special metrics for which this is not the case. Using Lie-theoretical methods, we present a unified approach to obtain all pairs (G, g) whose full isometry group \(\textrm{Isom}(G, g)\) has dimension greater than or equal to four. As a consequence, we determine, for every pair (G, g), up to automorphism and scaling, the dimension of \(\textrm{Isom}(G, g)\) , which can be three, four, or six.