A system of equations involving the fractional p-Laplacian and doubly critical nonlinearities

Abstract This article deals with existence of solutions to the following fractional p p -Laplacian system of equations: ( − Δ p ) s u = ∣ u ∣ p s * − 2 u + γ α p s * ∣ u ∣ α − 2 u ∣ v ∣ β in Ω , ( − Δ p ) s v = ∣ v ∣ p s * − 2 v + γ β p s * ∣ v ∣ β − 2 v ∣ u ∣ α in Ω , \left\{\begin{array}{l}{\left(-{\Delta }_{p})}^{s}u={| u| }^{{p}_{s}^{* }-2}u+\frac{\gamma \alpha }{{p}_{s}^{* }}{| u| }^{\alpha -2}u{| v| }^{\beta }\hspace{0.33em}\hspace{0.33em}\hspace{0.1em}\text{in}\hspace{0.1em}\hspace{0.33em}\Omega ,\hspace{1.0em}\\ {\left(-{\Delta }_{p})}^{s}v={| v| }^{{p}_{s}^{* }-2}v+\frac{\gamma \beta }{{p}_{s}^{* }}{| v| }^{\beta -2}v{| u| }^{\alpha }\hspace{0.33em}\hspace{0.33em}\hspace{0.1em}\text{in}\hspace{0.1em}\hspace{0.33em}\Omega ,\hspace{1.0em}\end{array}\right. where s ∈ ( 0 , 1 ) s\in \left(0,1) , p ∈ ( 1 , ∞ ) p\in \left(1,\infty ) with N > s p N\gt sp , α , β > 1 \alpha ,\beta \gt 1 such that α + β = p s * ≔ N p N − s p \alpha +\beta ={p}_{s}^{* }:= \frac{Np}{N-sp} and Ω = R N \Omega ={{\mathbb{R}}}^{N} or smooth bounded domains in R N {{\mathbb{R}}}^{N} . When Ω = R N \Omega ={{\mathbb{R}}}^{N} and γ = 1 \gamma =1 , we show that any ground state solution of the aforementioned system has the form ( λ U , τ λ V ) \left(\lambda U,\tau \lambda V) for certain τ > 0 \tau \gt 0 and U U and V V are two positive ground state solutions of ( − Δ p ) s u = ∣ u ∣ p s * − 2 u {\left(-{\Delta }_{p})}^{s}u={| u| }^{{p}_{s}^{* }-2}u in R N {{\mathbb{R}}}^{N} . For all γ > 0 \gamma \gt 0 , we establish existence of a positive radial solution to the aforementioned system in balls. When Ω = R N \Omega ={{\mathbb{R}}}^{N} , we also establish existence of positive radial solutions to the aforementioned system in various ranges of γ \gamma .