Nonexistence for Lane-Emden system involving Hardy potentials with singularities on the boundary

Our purpose of this article is to study nonexistence of positive super solutions for Lane-Emden system involving inverse-square potentials

$$\begin{aligned} -\Delta u+\frac{\mu _1}{|x|^2} u= v^p \ \ \textrm{in}\ \, \Omega ,\qquad -\Delta v+\frac{\mu _2}{|x|^2} v= u^q \ \ \textrm{in}\ \, \Omega , \end{aligned}$$(0.1)

where \(p,q>0\), \(\mu _1,\mu _2\ge -N^2/4\), \(\Omega \) is a bounded smooth domain in \(\mathbb {R}^N\) with \(N\ge 3\) such that \(0\in \partial \Omega \) and \(B^+_2(0):=\{x=(x',x_N)\in \mathbb {R}^{N-1}\times \mathbb {R}: x_N>0,\, |x|<2\}\subset \Omega \). Sharp critical curves of (q, p) are derived for nonexistence of positive super solutions to system (0.1) in the case that \(-N^2/4\le \mu _1,\mu _2<1-N\) and \(-N^2/4\le \mu _1<1-N\le \mu _2\). Our method is to iterate an initial singularities at the origin to improve the blowing-up rate until the nonlinearities are not admissible in some weighted \(L^1\) space.