System reliability assessment is one of the main activities in the operation and maintenance of the industrial sector. If the reliability assessment is inaccurate, it may cause wrong guidance for system maintenance. Although some progress has been made in system reliability assessment, the heavy reliance on data quality and the presence of multiple subsystem state dependencies mean that current purely data-driven methods are unable to fully address these challenges, resulting in limitations in achieving accurate reliability evaluations. In order to improve the accuracy of dynamic system reliability assessment, this paper proposes a hybrid system reliability assessment method that combines gradient boosting decision tree (GBDT) and dynamic Bayesian network (DBN). First, the data generation simulation based on the failure mechanism model is combined with the state diagnosis of the GBDT. Then, monitoring nodes for key components are added to the DBN, and the GBDT is used to establish a mapping relationship between monitoring data and components states. The physical mapping relationships provide objective information, unlike the subjective factors resulting from relying solely on expert experience. The DBN integrates component dependent relationships and monitoring nodes of components to evaluate the system operational reliability. A harmonic gear drive (HGD) system is taken as a case study to verify the proposed method. The results show that the proposed method reduces the relative error percentage in operational reliability assessment by 33 %.