Many people suffer from pollen allergies. Therefore, pollen monitoring is performed worldwide, and pollen traps are used for this purpose. Specialists are analyzing the acquired material, using adequate reference material for the identification of target taxa. However, the background in reference images is relatively clean and uniform, whereas the background in pollen trap images contains air bubbles, dust, fungal and fern spores, and pollen grains of various taxa. In this work, we address the automatic detection and identification of the pollen grains of selected tree species, responsible for most common allergies, in images from pollen traps. Deep neural networks have been applied for this purpose, using models trained on reference pollen grain images. These models produce unsatisfactory results when applied to images from pollen traps. Problems related to the background in these images are discussed in this paper, and fine-tuning of the models for the recognition of allergenic pollen grains is presented. The obtained results are discussed, and future works are indicated.