Multiple autonomous underwater vehicles (AUVs) target tracking problem is a significant challenge for AUV swarm control, which is crucial to the growth of the marine industry. To emphasize the great adaptability while tackling the limitations of reinforcement learning (RL) methods in Multi-AUV target tracking tasks, we propose an efficient two-stage learning from demonstrations (LfD) training framework, FISHER, based on few-shot expert demonstration, featuring imitation learning (IL) and offline reinforcement learning (ORL). In the first stage, we develop a sample-efficient algorithm, multi-agent discriminator actor-critic (MADAC), to facilitate the imitation of expert policy and the generation of offline datasets. In the second stage, based on the decision transformer (DT), the reward function-independent algorithm, multi-agent independent generalized decision transformer (MAIGDT) is utilized for further policy improvement. Simultaneously, we propose a simulation to simulation (sim2sim) method to facilitate the generation of expert trajectories, which is compatible with traditional methods like artificial potential field (APF). Through comparative experiments, we verify the improvement of the proposed MADAC and MAIGDT algorithms. Finally, full target tracking simulation processes show that FISHER can achkmieve performance comparable to expert demonstrations, thereby further demonstrating the strong practicality of FISHER framework. To accelerate relevant research in this direction, the code for simulation will be released as open-source.