Background <p>Introgressive hybridization is common in natural birch woodlands in Iceland, where two birch (<i>Betula</i>) species (diploid dwarf birch <i>B. nana</i> and tetraploid tree birch <i>B. pubescens</i>) coexist and hybridize readily. Our previous morphological, cytogenetic and palynological studies show that triploid hybrids are likely to have mediated gene flow between the two species. Our previous molecular study based on chloroplast haplotyping confirms the hybrid introgression and provides information about the genetic origin of <i>Betula</i> species in Iceland. The question remains, however, as to what extent nuclear gene flow is involved in this hybrid introgression process. The objective of the present study was therefore to use nuclear markers to probe birch introgressive hybridization.</p> Results <p>AFLP (Amplified Fragment Length Polymorphism) analysis was performed on genomic DNA isolated from 169 individual <i>Betula</i> plants (67 diploid <i>B. nana</i>, 82 tetraploid <i>B. pubescens</i> and 20 triploid hybrids), from birch woodlands in Iceland in comparison to those from northern Scandinavia. The generated 115 polymorphic markers were subjected to analysis of molecular variance across ploidy groups, locations, and major chloroplast haplotypes. A new R package, Linarius, was developed for use with this mixed ploidy dataset. All markers were considered nuclear as no allele specific to any chloroplast haplotypes was detected. The results were to a certain extent congruent with those from our previous chloroplast study. No ploidy- or species-specific alleles were detected. Almost all alleles were shared among all three ploidy groups, indicating gene flow via hybridization. The difference, however, was that the nuclear markers clearly differentiated between diploid <i>B. nana</i> and tetraploid <i>B. pubescens</i>, whereas the chloroplast haplotype variation between species was non-significant. The triploid hybrid group was scattered within both ploidy clusters, in line with its role as a bridge to introgression. This nuclear separation between the two species is comparable to that from our previous analysis based on species- specific morphological characters, implying that the whole genomes may be selected for species adaptability in their different habitats. Furthermore, the present AFLP study depicted a clear east–west geographical separation among Icelandic <i>Betula</i> populations, based on both genetic distance analysis and anamorphosis modelling. This geographical separation is prominent in <i>B. nana</i> while <i>B. pubescens</i> is more genetically homogeneous.</p> Conclusion <p>The present study shows that despite extensive gene flow, <i>Betula</i> species maintain their species integrity and ploidy stability. This in turn allows the long-term survival of the species in their local habitats.</p>