In this paper, we consider the following fractional Hamiltonian system, \(\begin{aligned} \begin{aligned} {\left\{ \begin{array}{ll} (-\Delta )^{s}u = g(x,v) & \text{ in } \Omega ,\\ (-\Delta )^{s}v = f(x,u) & \text{ in } \Omega ,\\ u\ge 0, v\ge 0 \quad & \text{ in } \Omega ,\\ u=v=0 \quad & \text{ in } {\mathbb {R}}^{N}\setminus \Omega , \end{array}\right. } \end{aligned} \end{aligned}\) where \(s\in (0,1),\) \(N>2s,\) \(\Omega \subset {\mathbb {R}}^N\) is a smooth bounded domain and \(f,g:\overline{\Omega } \times {\mathbb {R}} \rightarrow {\mathbb {R}}\) are continuous functions. By variational method and fractional Orlicz-Sobolev spaces, we demonstrate the existence of nontrivial weak solutions. As far as we know, there are few research works devoted to the fractional Hamiltonian system on a bounded domain. Our work exhibits threefold of novelties. Firstly, we deal with a non-autonomous system; the upper bound restriction required in previous works is no longer needed; moreover, we can treat almost critical nonlinearities (see (1.18) below).
