Constructing a long-term global dataset of direct and diffuse radiation (10 km, 3 h, 1983–2018) separating from the satellite-based estimates of global radiation

In addition to global radiation (R_g), direct radiation (R_dir) and diffuse radiation (R_dif) are important fundamental data urgently needed in scientific and industrial fields. However, compared with R_g, R_dir and R_dif have received little attention in the past, either in observations or in satellite retrievals, mainly due to the high cost of their observations and the difficulty of retrieving them effectively from satellites. In this study, a long-term global gridded dataset of R_dir and R_dif was constructed by separating from a high-precision satellite-based product of R_g using the Light Gradient Boosting Machine (LightGBM) model, trained with in-situ observations measured at the Baseline Surface Radiation Network (BSRN). The inputs to construct the dataset are the four variables of R_g, the cloud transmittance for R_g, the ratio of R_dif to R_g under clear sky condition (call the clear diffuse ratio), and the cosine of the solar zenith angle. The developed dataset was validated against in-situ observations and compared with other satellite-based products. Evaluations against the BSRN observations indicated that our proposed method has good generality and outperforms the machine learning-based direct estimation method of Hao et al. (2020). Independent validations were further performed against the observations measured at 17 China Meteorological Administration (CMA) radiation stations and the estimation based on sunshine duration observations at >2400 CMA routine meteorological stations, respectively. It was found that the accuracies of our estimates for both R_dir and R_dif were improved when upscaled to ≥ 30 km. Comparisons with three other satellite-based products indicate that our developed dataset of both R_dir and R_dif was generally more accurate than the global products of the Earth's Radiant Energy System (CERES) and Hao et al. (2020) based on the Deep Space Climate Observatory/Earth Polychromatic Imaging Camera (EPIC) (DSCOVER/EPIC) satellite, and the regional gridded product (JIEA) of Jiang et al. (2020a). The dataset developed in this study will contribute to ecological research and solar engineering applications.