The thermal diffusivity of magnetocaloric materials has a transition point at a given temperature that depends on the intensity of the applied magnetic field. Consequently, a fine temperature resolution on the material sample is needed to obtain an accurate determination of the thermal diffusivity variation with temperature. The coupling between the external and the internal fields has to be carefully mastered since the pertinent operating condition is fixed by the actual internal field which is not directly measurable and may be heavily affected by any element in contact with the sample. Therefore, contactless methods such as Photo-Thermal Radiometry (PTR) are privileged. The latter is based on a radiative excitation of the front face of a thin sample and the detection of the thermal effect on the opposite face. However, the powerful radiative source may significantly increase the sample temperature which is not suitable for caloric materials. In this work, a low power modulated PTR method is proposed to characterize second order magnetocaloric materials under magnetic field. It was compared to high energy thermo-flash PTR and validated on common materials such as steel and stainless steel, and then applied to gadolinium which is the reference magnetocaloric material for magnetic refrigeration and heat pumping study. The thermal diffusivity of gadolinium samples is measured in the 285.1 K to 305.1 K temperature range, including the magnetic transition temperature without and under an external magnetic flux density of 0.5 T in the 13 mm air gap of the permanent magnet magnetic circuit. The low power probe beam ensures a temperature stability with a negligible sample temperature fluctuation less than 0.05 K on the incident sample surface and less than 0.03 K on the measurement surface. The experimental results without magnetic field align with those using other methods including the magnetic transition temperatures determination. This low-power optical method proved its efficiency to characterize highly temperature dependent materials such as magnetocaloric materials sensitive to magnetic field. The data obtained partly fills the lack of information in the literature on excited gadolinium.