SiGe/SiO2 multilayers with layer thickness of 5 nm were deposited with RF magnetron sputtering. The as deposited samples had well defined SiGe amorphous layers. Different annealing treatments were made to promote the formation of SiGe nanocrystals. We report an ion beam analysis study with the Rutherford backscattering and elastic recoil analysis detection techniques, in order to determine the thickness and composition of the nanolayers, and gain insight into the evolution of the roughness of the layers. The results are correlated with other structural properties of the samples, as measured with complementary techniques such as grazing incidence X-ray diffraction annular dark field scanning transmission electron microscopy and high resolution transmission electron microscopy.