We demonstrate a hybrid nanocomposite combining mesoporous silica, p \(\hbox {SiO}_2\) , as a host medium and guest lithium niobate \(\hbox {LiNbO}_3\) nanocrystals embedded into tubular silica nanochannels by calcination of the precursor mixed solution of lithium and niobium salts. High-resolution transmission electron microscopy, X-ray diffraction and Raman scattering techniques reveal trigonal \(\hbox {LiNbO}_3\) nanocrystals within the p \(\hbox {SiO}_2\) nanochannels, indicating their random texture morphology. Annealing at high temperatures ( 950 \(^{\circ }\) C) during calcination also leads to partial crystallization of the p \(\hbox {SiO}_2\) matrix with the formation of trigonal \(\alpha\) - \(\hbox {SiO}_2\) nanocrystals. The Raman microscopy analysis of the p \(\hbox {SiO}_2\) : \(\hbox {LiNbO}_3\) nanocomposite reveals three structural crystalline phases, \(\alpha -\hbox {SiO}_2\) , \(\hbox {LiNbO}_3\) and a mixed phase which involves the \(\alpha\) - \(\hbox {SiO}_2\) phase of host membrane and \(\hbox {LiNbO}_3\) nanocrystals embedded into the membrane. The finite size of the \(\hbox {LiNbO}_3\) nanocrystals results in specific features of the LO-TO phonon frequency splitting, which are investigated by Raman microscopy. In the transmission geometry, the second harmonic generation emission exhibits no Maker fringes and is characterized by a broad angular diagram of diffusely scattered light. The second harmonic generation response is independent of the polarization direction of the incident pump light, thus indicating a spatial isotropy of the nonlinear optical conversion in the p \(\hbox {SiO}_2\) : \(\hbox {LiNbO}_3\) composite, consistent with the randomly oriented textural morphology of the deposited \(\hbox {LiNbO}_3\) nanocrystals. The contribution of the guest \(\hbox {LiNbO}_3\) nanocrystals to the second harmonic generation effect was found to be strongly dominant compared to the partially crystallized host p \(\hbox {SiO}_2\) matrix. The nanocomposite p \(\hbox {SiO}_2\) : \(\hbox {LiNbO}_3\) membrane, set in the 90 \(^\circ\) nonlinear optical geometry, shows unusually high diffusely transmitted second harmonic generation light (back-reflected emission), apparently supported by internal light reflection from the tubular nanochannel network. Despite the fundamental interest, the revealed anomalous back-reflected second harmonic generation emission from p \(\hbox {SiO}_2\) : \(\hbox {LiNbO}_3\) nanocomposite membranes expands the prospects for their photonic and nonlinear optical applications.
