MGTNSyn: Molecular structure-aware graph transformer network with relational attention for drug synergy prediction

Abstract

Accurately predicting the synergistic effects of drug combinations is a significant challenge for modern personalized oncology treatments. Graph neural networks (GNNs) can capture rich structural information about drug molecules, supporting the prediction of cancer drug responses and accelerating the discovery of novel drug combinations. However, the existing GNN-based methods have problems such as over-smoothing and over-squashing, which limit the techniques’ ability to express the structural information of drug molecules. To this end, this study proposes a molecular structure-aware graph Transformer network with relational attention for predicting drug synergy (MGTNSyn). MGTNSyn utilizes a graph relational attention network to aggregate key local substructures and identify molecule functional groups. It also employs the molecular structure-aware graph Transformer network to detect mutagenic motifs in drugs from a global perspective. The information on drug structure obtained at local and global levels enables MGTNSyn to better understand the mechanism of drug therapy for cancer. Extensive experiments on two real-world datasets demonstrate that MGTNSyn outperforms other state-of-the-art methods and alleviates expression limitations. The novel drug combination prediction experiments on three cancer cell lines demonstrate the method’s ability to discover therapeutic drug combinations.