Existence of nontrivial solutions for a fractional \(p\&q\)-Laplacian equation with sandwich-type and sign-changing nonlinearities

In this paper, we deal with the following fractional $p\mathrm{\&}q$ $p\&q$ -Laplacian problem: $\{\begin{array}{ll}{(-\mathrm{\Delta })}_p^{s}u+{(-\mathrm{\Delta })}_q^{s}u=\lambda a(x){|u|}^{\theta -2}u+\mu b(x){|u|}^{r-2}u& \text{in}\mathrm{\Omega },\\ u(x)=0& \text{in}{\mathbb{R}}^N\setminus \mathrm{\Omega },\end{array}$ \( \left \{ \textstyle\begin{array}{l@{\quad }l} (-\Delta )_{p}^{s}u +(-\Delta )_{q}^{s}u =\lambda a(x)|u|^{\theta -2}u+ \mu b(x)|u|^{r-2}u&\text{in}\;\ \Omega , \\ u(x)=0 &\text{in}\;\ \mathbb{R}^{N}\setminus \Omega , \end{array}\displaystyle \right . \) where $\mathrm{\Omega }\subset {\mathbb{R}}^N$ $\Omega \subset \mathbb{R}^{N}$ is a bounded domain with smooth boundary, $s\in (0,1)$ $s\in (0,1)$ , ${(-\mathrm{\Delta })}_m^s$ $(-\Delta )_{m}^{s}$ $(m\in \{p,q\})$ $(m\in \{p,q\})$ is the fractional m-Laplacian operator, $p,q,r,\theta \in (1,p_s^{\ast }]$ $p,q,r,\theta \in (1,p_{s}^{*}]$ , $p_s^{\ast }=\frac{Np}{N-sp}$ $p_{s}^{*}=\frac{Np}{N-sp}$ , $\lambda ,\mu >0$ $\lambda , \mu >0$ , and the weights $a(x)$ $a(x)$ and  $b(x)$ $b(x)$ are possibly sign changing. Using the concentration compactness principle for fractional Sobolev spaces and the Ekeland variational principle, we prove that the problem admits a nonnegative solution for the critical case $r=p_s^{\ast }$ $r=p_{s}^{*}$ . Moreover, for the subcritical case $r<p_s^{\ast }$ $r< p_{s}^{*}$ , we obtain two existence results by applying the Ekeland variational principle and the mountain pass theorem.