Infrared small target detection (IRSTD) is a key component of infrared target tracking and search. This task is quite challenging due to the small and dim infrared targets, complex backgrounds, and low target-background contrast. Current infrared remote sensing detection algorithms exhibit performance that is strongly dependent on data scale and quality, yet this performance is constrained by the scarcity of publicly available datasets. Moreover, most existing methods are trained only on a single style domain, resulting in poor generalization to cross-domain scenarios. To address these issues, we propose E2SAM, an Edge-Enhanced Segment Anything Model tailored for IRSTD. E2SAM adopts a dual-branch design: one branch leverages the Segment Anything Model (SAM) for mask prediction, while the other utilizes lightweight convolution for edge detection. We further introduce a fast Fourier convolution adapter to extract high-frequency features. By embedding physical priors such as high-frequency and edge information, E2SAM guides the base model to extract target features more accurately, mitigating style domain bias. To evaluate its few-shot learning capability, we conduct experiments with limited training data (using 30%, 50%, and 80% of the original datasets), and the results show that E2SAM maintains excellent detection performance under such conditions. It also generalizes well in cross-domain scenarios, validating the effectiveness of the proposed method.