{"title":"自适应稀疏变压器Hawkes过程","authors":"Yue Gao, Jian-Wei Liu","doi":"10.1142/s0218488523500319","DOIUrl":null,"url":null,"abstract":"Nowadays, many sequences of events are generated in areas as diverse as healthcare, finance, and social network. People have been studying these data for a long time. They hope to predict the type and occurrence time of the next event by using relationships among events in the data. recently, with the successful application of Recurrent Neural Network (RNN) in natural language processing, it has been introduced into point process. However, RNN cannot capture the long-term dependence among events well, and self-attention can partially mitigate this problem precisely. Transformer Hawkes Process (THP) using self-attention greatly improves the performance of the Hawkes Process, but THP cannot ignore the effect of irrelevant events, which will affect the computational complexity and prediction accuracy of the model. In this paper, we propose an Adaptively Sparse Transformers Hawkes Process (ASTHP). ASTHP considers the periodicity and nonlinearity of event time in the time encoding process. The sparsity of the ASTHP is achieved by substituting Softmax with [Formula: see text]-entmax: [Formula: see text]-entmax is a differentiable generalization of Softmax that allows unrelated events to gain exact zero weight. By optimizing the neural network parameters, different attention heads can adaptively select sparse modes (from Softmax to Sparsemax). Compared with the existing models, ASTHP model not only ensures the prediction performance but also improves the interpretability of the model. For example, the accuracy of ASTHP model on MIMIC-II dataset is improved by nearly 3 percentage points, and the model fitting degree and stability are also improved significantly.","PeriodicalId":50283,"journal":{"name":"International Journal of Uncertainty Fuzziness and Knowledge-Based Systems","volume":"19 1","pages":""},"PeriodicalIF":1.0000,"publicationDate":"2023-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Adaptively Sparse Transformers Hawkes Process\",\"authors\":\"Yue Gao, Jian-Wei Liu\",\"doi\":\"10.1142/s0218488523500319\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Nowadays, many sequences of events are generated in areas as diverse as healthcare, finance, and social network. People have been studying these data for a long time. They hope to predict the type and occurrence time of the next event by using relationships among events in the data. recently, with the successful application of Recurrent Neural Network (RNN) in natural language processing, it has been introduced into point process. However, RNN cannot capture the long-term dependence among events well, and self-attention can partially mitigate this problem precisely. Transformer Hawkes Process (THP) using self-attention greatly improves the performance of the Hawkes Process, but THP cannot ignore the effect of irrelevant events, which will affect the computational complexity and prediction accuracy of the model. In this paper, we propose an Adaptively Sparse Transformers Hawkes Process (ASTHP). ASTHP considers the periodicity and nonlinearity of event time in the time encoding process. The sparsity of the ASTHP is achieved by substituting Softmax with [Formula: see text]-entmax: [Formula: see text]-entmax is a differentiable generalization of Softmax that allows unrelated events to gain exact zero weight. By optimizing the neural network parameters, different attention heads can adaptively select sparse modes (from Softmax to Sparsemax). Compared with the existing models, ASTHP model not only ensures the prediction performance but also improves the interpretability of the model. For example, the accuracy of ASTHP model on MIMIC-II dataset is improved by nearly 3 percentage points, and the model fitting degree and stability are also improved significantly.\",\"PeriodicalId\":50283,\"journal\":{\"name\":\"International Journal of Uncertainty Fuzziness and Knowledge-Based Systems\",\"volume\":\"19 1\",\"pages\":\"\"},\"PeriodicalIF\":1.0000,\"publicationDate\":\"2023-08-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"International Journal of Uncertainty Fuzziness and Knowledge-Based Systems\",\"FirstCategoryId\":\"94\",\"ListUrlMain\":\"https://doi.org/10.1142/s0218488523500319\",\"RegionNum\":4,\"RegionCategory\":\"计算机科学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q4\",\"JCRName\":\"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"International Journal of Uncertainty Fuzziness and Knowledge-Based Systems","FirstCategoryId":"94","ListUrlMain":"https://doi.org/10.1142/s0218488523500319","RegionNum":4,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q4","JCRName":"COMPUTER SCIENCE, ARTIFICIAL INTELLIGENCE","Score":null,"Total":0}
Nowadays, many sequences of events are generated in areas as diverse as healthcare, finance, and social network. People have been studying these data for a long time. They hope to predict the type and occurrence time of the next event by using relationships among events in the data. recently, with the successful application of Recurrent Neural Network (RNN) in natural language processing, it has been introduced into point process. However, RNN cannot capture the long-term dependence among events well, and self-attention can partially mitigate this problem precisely. Transformer Hawkes Process (THP) using self-attention greatly improves the performance of the Hawkes Process, but THP cannot ignore the effect of irrelevant events, which will affect the computational complexity and prediction accuracy of the model. In this paper, we propose an Adaptively Sparse Transformers Hawkes Process (ASTHP). ASTHP considers the periodicity and nonlinearity of event time in the time encoding process. The sparsity of the ASTHP is achieved by substituting Softmax with [Formula: see text]-entmax: [Formula: see text]-entmax is a differentiable generalization of Softmax that allows unrelated events to gain exact zero weight. By optimizing the neural network parameters, different attention heads can adaptively select sparse modes (from Softmax to Sparsemax). Compared with the existing models, ASTHP model not only ensures the prediction performance but also improves the interpretability of the model. For example, the accuracy of ASTHP model on MIMIC-II dataset is improved by nearly 3 percentage points, and the model fitting degree and stability are also improved significantly.
期刊介绍:
The International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems is a forum for research on various methodologies for the management of imprecise, vague, uncertain or incomplete information. The aim of the journal is to promote theoretical or methodological works dealing with all kinds of methods to represent and manipulate imperfectly described pieces of knowledge, excluding results on pure mathematics or simple applications of existing theoretical results. It is published bimonthly, with worldwide distribution to researchers, engineers, decision-makers, and educators.