Arsenic and selenium are toxic semimetals, and removal of these semimetals from environmental water is a global problem to be solved. This study modified chitosan nanofiber using xylose (XY), which has multiple hydroxyl groups with high affinity to semimetals as ligands to enhance semimetals adsorption. Adsorption properties of semimetals by cross-linked chitosan nanofiber derivative (CNF-XY) were compared with cross-linked chitosan nanofiber (CNF) and commercially synthesized and commercially available N-methylglucamine-type chelating resins (CRB05, GRY-HW). Se(IV) and Se(VI) adsorption by CNF-XY and CNF was 2.8 to 8.8 times faster than that by commercial resins, and the maximum adsorption capacity was 2.3 to 10 times larger than that by commercial resins. The adsorption kinetics of arsenic and selenium were analyzed using pseudo-first-order, pseudo-second-order, and Weber-Morris intraparticle diffusion models. The adsorption of As(III) and As(V) by CNF-XY was faster than those by commercial resins. The adsorption of As(III) and As(V) by CNF-XY fit the pseudo-second-order kinetic model but did not pass the origin, suggesting that intraparticle diffusion is responsible for As(III), As(V) adsorption was not the only rate-limiting step controlling As(III), As(V) adsorption, indicating some control by boundary membrane diffusion, suggesting that multiple processes were affected by adsorption. Se(IV) and Se(VI) adsorption onto CNF-XY and CNF are predominant by the pseudo-first-order adsorption mechanism.