The Brezis–Nirenberg Problem for Fractional p-Laplacian Systems in Unbounded Domains

In this article we consider the following fractional p-Laplacian system \(\begin{aligned} {\left\{ \begin{array}{ll} (-\Delta _{p})^{s}u=Q_{u}(u,v)+H_{u}(u,v) \ \ \ {} &{} {\text{ in }} \ \Omega , \\ (-\Delta _{p})^{s}v=Q_{v}(u,v)+H_{v}(u,v) \ \ \ {} &{} {\text{ in }} \ \Omega , \\ ~u,v\ge 0,~~u,v\ne 0, \ \ \ {} &{} {\text{ in }} \ \Omega , \\ ~u=v=0 \ \ \ {} &{} {\text{ in }} \ \mathbb {R}^{N}\backslash \Omega , \end{array}\right. } \end{aligned}\) $$\begin{array}{c}\hfill \left\{\begin{array}{cc}{(-{\mathrm{\Delta }}_p)}^{s}u=Q_u(u,v)+H_u(u,v)\hfill & \text{in}\mathrm{\Omega },\hfill \\ {(-{\mathrm{\Delta }}_p)}^{s}v=Q_v(u,v)+H_v(u,v)\hfill & \text{in}\mathrm{\Omega },\hfill \\ u,v\ge 0,u,v\ne 0,\hfill & \text{in}\mathrm{\Omega },\hfill \\ u=v=0\hfill & \text{in}{\mathbb{R}}^N\backslash \mathrm{\Omega },\hfill \end{array}\right)\end{array}$$ where \(\Omega \subset \mathbb {R}^{N}\) $\mathrm{\Omega }\subset {\mathbb{R}}^N$ is an unbounded strip like domain, \(s\in (0,1)\) $s\in (0,1)$ , \(p>1\) $p>1$ and \(ps<N\) $ps<N$ , \(p_{s}^{*}=\frac{Np}{N-ps}\) $p_s^{\ast }=\frac{\mathit{Np}}{N-ps}$ , Q, H are homogeneous functions of degrees p and \(p_{s}^{*}\) $p_s^{\ast }$ , respectively. By means of the fractional p-Poincaré inequality in infinite cylindrical domains, we prove the existence of nontrivial weak solutions for the above system through variational techniques. The present work extends some known Brézis–Nirenberg type results to the fractional p-Laplacian on unbounded domains.