We study algebras of differential and difference operators acting on matrix valued orthogonal polynomials (MVOPs) with respect to a weight matrix of the form \(W^{(\nu )}_{\phi }(x) = x^{\nu }e^{-\phi (x)} W^{(\nu )}_\textrm{pol}(x)\) , where \(\nu >0\) , \(W^{(\nu )}_\textrm{pol}(x)\) is a certain matrix valued polynomial and \(\phi \) is an analytic function. We introduce differential operators \({\mathcal {D}}\) , \({\mathcal {D}}^{\dagger }\) which are mutually adjoint with respect to the matrix inner product induced by \(W^{(\nu )}_{\phi }(x)\) . We prove that the Lie algebra generated by \({\mathcal {D}}\) and \({\mathcal {D}}^{\dagger }\) is finite dimensional if and only if \(\phi \) is a polynomial. For a polynomial \(\phi \) , we describe the structure of this Lie algebra. As a byproduct, we give a partial answer to a problem by Ismail about finite dimensional Lie algebras related to scalar Laguerre type polynomials. The case \(\phi (x)=x\) is discussed in detail.