Jorgensen and Pedersen (J. Anal. Math., 1998) showed that the first example of middle-third Cantor measure, referred to as a non-spectral measure, does not admit any orthogonal \(L^2\) Fourier series, and there exist at most two orthogonal exponential functions in \(L^2\) -space. For a non-spectral measure \(\mu \) , we typically focus on twofold questions: (C1) whether \(L^2(\mu )\) contains only finitely many orthogonal exponential functions; (C2) whether \(L^2(\mu )\) admits infinite families of orthogonal exponential functions, none of which form an orthogonal basis. In this work, based on some techniques from matrix theory over finite fields, we study non-spectral problem concerning a class of iterated function system (IFS) measures \(\mu _{M,D}\) in \(\mathbb {R}^2\) , where \(M\in M_{2}(\mathbb {Z})\) is expanding, and the integer digit set \(D=\{(0,0)^t,(1,0)^t,(0,1)^t,(d,d)^t\}\) with \(d\ne 1\) . This paper provides an almost comprehensive investigation of this problem, and the exact maximal cardinality is given in term of question (C1).