Denote by \(T_n^d(A)\) an upper triangular operator matrix of dimension \(n\in \mathbb {N}\) whose diagonal entries \(D_i,\ 1\le i\le n\) , are known, and \(A=(A_{ij})_{1\le i<j\le n}\) is an unknown tuple of operators. This article is aimed at investigation of defect spectrum \(\mathcal {D}^{\sigma _*}=\bigcup _{i=1}^n\sigma _*(D_i){\setminus }\sigma _*(T_n^d(A))\) , where \(\sigma _*\) is a spectrum corresponding to various types of invertibility: (left, right) invertibility, (left, right) Fredholm invertibility, left/right Weyl invertibility. We give characterizations for each of the previous types, and provide some sufficent conditions for the stability of certain spectrum (the case \(\mathcal {D}^{\sigma _*}=\emptyset \) ). The results are proved for all matrix dimensions \(n\ge 2\) , and they hold in arbitrary Hilbert spaces without assuming separability, thus generalizing results from Wu and Huang (Ann Funct Anal 11(3):780–798, 2020; Acta Math Sin 36(7):783–796, 2020). We also retrieve a result from Bai et al. (J Math Anal Appl 434(2):1065–1076, 2016) in the case \(n=2\) , and we provide a precise form of the well known ‘filling in holes’ result from Han et al. (Proc Am Math Soc 128(1):119–123, 2000).