A novel physics-based cloud retrieval algorithm based on neural networks (CRANN) from hyperspectral measurements in the O2-O2 band

Clouds play a crucial role in the Earth's climate system and their properties can be detected by hyperspectral measurements from space. With the increasing spectral resolution, traditional retrieval methods based on look-up tables (LUT) and optimal estimation are limited in both efficiency and accuracy compared with machine learning methods. However, the machine learning techniques used to establish the relationships between spectral measurements and cloud properties often lack physical explainability and universality. Additionally, traditional physical retrieval methods based on oxygen A-band are not applicable to instruments without the O₂-A band like the ozone monitoring instrument (OMI). Therefore, we have proposed a novel physics-based deep neural networks (DNN) retrieval method––the cloud retrieval algorithm based on neural networks (CRANN)––which incorporates a deep neural network model with radiative transfer model to retrieve cloud fraction and cloud-top pressure from the oxygen–oxygen collision-induced (O₄) absorption band. Validation using simulated test data supported the superior accuracy of CRANN, with the correlation coefficients for cloud fraction and cloud-top pressure are 0.989 and 0.993, respectively, whereas the correlation coefficients for cloud fraction and cloud-top pressure of the LUT method are 0.928 and 0.865, respectively. In comparison with the OMCLDO2 cloud product from the OMI, the CRANN results retrieved from OMI observations exhibit substantial consistency, boasting correlation coefficients surpassing 0.95 for cloud fraction and 0.83 for cloud pressure. As compared with the tropospheric monitoring instrument (TROPOMI) official products, the CRANN retrieval results from TROPOMI exhibit a high level of consistency with correlation coefficients exceeding 0.8 for cloud fraction and 0.73 for cloud pressure. Additionally, a promising agreement is observed between the CRANN retrievals from TROPOMI and Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP) data, yielding RMSEs of 127.3, 134.6 and 106.4 hPa for the validation dataset, respectively.