Multi-task optimization (MTO) improves computational efficiency and solution performance by processing multiple tasks simultaneously and leveraging knowledge transfer. The multifactorial evolutionary algorithm (MFEA) was a pioneering MTO method. Its core concept utilizes parallel population search and cross-task knowledge sharing to optimize multiple tasks concurrently. However, MFEA does not differentiate between crossover operations for individuals with identical skill factors and those with different skill factors. To solve this problem, this paper proposes an improved multi-factor optimization algorithm (AEDA-MFEA), where the algorithm sets an elite diversity archive for each task, aiming to more fully utilize the information within the population; and sets an adaptive crossover probability and utilizes affine transformation to generate the crossover individuals for individuals with different tasks, which makes it easier for the population to obtain favorable migration knowledge. Experiments show that AEDA-MFEA has excellent performance when dealing with both simple and complex problems with dual tasks.