When a sound source that can interfere with concentration exists in the same direction as a user’s focused task, it can lead to problems such as reduced ease of work and increased workload. This study addresses situations where a distracting sound source is present in the same direction as the user’s focused task and proposes a method to improve the ease of performing the task. The proposed method uses AR technology to virtually move only the visual information of the distracting sound source (e.g., the appearance of people conversing) to a direction different from the user’s focused task. This aims to make the user perceive the distracting sound source as being at that virtual location, thereby improving the ease of task performance. The results of an evaluation experiment confirmed that the proposed method has the effect of significantly reducing the workload of a cognitive task. These findings aid the design of AR-based task concentration support systems.