Pub Date : 2023-05-01DOI: 10.1109/InteNSE59150.2023.00006
Razan Baltaji, Parth Thakkar
Machine learning techniques have found a widespread use in the software engineering community. In particular, language models (LMs) trained on code form the backbone of a majority of these applications, spanning tasks such as code completion, summarization, refactoring, execution prediction, and test generation. These tasks require reasoning about both the syntax and semantics of code. Recent work has shown that language models learn to capture the syntactic properties of code, but it is unclear to what extent they can reason about the semantics of code. In this work, we explore the ability of 3 language models of code to reason about a specific kind of semantics: numerical and logical properties of code. We propose several probing tasks to test the numerical and logical reasoning abilities of these models. We find that the models we explore - CodeBERT, GraphCodeBERT and CodeGen do indeed learn many numerical and logical properties of code, such as finding maximum in a list of numbers, comparing numbers, evaluating boolean expressions and representing numbers. They do not perform as well on complex tasks such as evaluating arithmetic expressions and substituting variables in such expressions. Our results indicate that while these models hold promise, there is a lot of room for improvement of their numeric and logical reasoning abilities.
{"title":"Probing Numeracy and Logic of Language Models of Code","authors":"Razan Baltaji, Parth Thakkar","doi":"10.1109/InteNSE59150.2023.00006","DOIUrl":"https://doi.org/10.1109/InteNSE59150.2023.00006","url":null,"abstract":"Machine learning techniques have found a widespread use in the software engineering community. In particular, language models (LMs) trained on code form the backbone of a majority of these applications, spanning tasks such as code completion, summarization, refactoring, execution prediction, and test generation. These tasks require reasoning about both the syntax and semantics of code. Recent work has shown that language models learn to capture the syntactic properties of code, but it is unclear to what extent they can reason about the semantics of code. In this work, we explore the ability of 3 language models of code to reason about a specific kind of semantics: numerical and logical properties of code. We propose several probing tasks to test the numerical and logical reasoning abilities of these models. We find that the models we explore - CodeBERT, GraphCodeBERT and CodeGen do indeed learn many numerical and logical properties of code, such as finding maximum in a list of numbers, comparing numbers, evaluating boolean expressions and representing numbers. They do not perform as well on complex tasks such as evaluating arithmetic expressions and substituting variables in such expressions. Our results indicate that while these models hold promise, there is a lot of room for improvement of their numeric and logical reasoning abilities.","PeriodicalId":166762,"journal":{"name":"2023 IEEE/ACM International Workshop on Interpretability and Robustness in Neural Software Engineering (InteNSE)","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133402592","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-03-08DOI: 10.1109/InteNSE59150.2023.00007
Aftab Hussain, Md Rafiqul Islam Rabin, Bowen Xu, David Lo, Mohammad Amin Alipour
Although deep neural models substantially reduce the overhead of feature engineering, the features readily available in the inputs might significantly impact training cost and the performance of the models. In this paper, we explore the impact of an unsuperivsed feature enrichment approach based on variable roles on the performance of neural models of code. The notion of variable roles (as introduced in the works of Sajaniemi et al. [1], [2]) has been found to help students' abilities in programming. In this paper, we investigate if this notion would improve the performance of neural models of code. To the best of our knowledge, this is the first work to investigate how Sajaniemi et al.'s concept of variable roles can affect neural models of code. In particular, we enrich a source code dataset by adding the role of individual variables in the dataset programs, and thereby conduct a study on the impact of variable role enrichment in training the Code2Seq model. In addition, we shed light on some challenges and opportunities in feature enrichment for neural code intelligence models.
{"title":"A Study of Variable-Role-based Feature Enrichment in Neural Models of Code","authors":"Aftab Hussain, Md Rafiqul Islam Rabin, Bowen Xu, David Lo, Mohammad Amin Alipour","doi":"10.1109/InteNSE59150.2023.00007","DOIUrl":"https://doi.org/10.1109/InteNSE59150.2023.00007","url":null,"abstract":"Although deep neural models substantially reduce the overhead of feature engineering, the features readily available in the inputs might significantly impact training cost and the performance of the models. In this paper, we explore the impact of an unsuperivsed feature enrichment approach based on variable roles on the performance of neural models of code. The notion of variable roles (as introduced in the works of Sajaniemi et al. [1], [2]) has been found to help students' abilities in programming. In this paper, we investigate if this notion would improve the performance of neural models of code. To the best of our knowledge, this is the first work to investigate how Sajaniemi et al.'s concept of variable roles can affect neural models of code. In particular, we enrich a source code dataset by adding the role of individual variables in the dataset programs, and thereby conduct a study on the impact of variable role enrichment in training the Code2Seq model. In addition, we shed light on some challenges and opportunities in feature enrichment for neural code intelligence models.","PeriodicalId":166762,"journal":{"name":"2023 IEEE/ACM International Workshop on Interpretability and Robustness in Neural Software Engineering (InteNSE)","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130075659","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-03-03DOI: 10.1109/InteNSE59150.2023.00005
Md Rafiqul Islam Rabin, Aftab Hussain, Sahil Suneja, Mohammad Amin Alipour
Finding important features that contribute to the prediction of neural models is an active area of research in explainable AI. Neural models are opaque and finding such features sheds light on a better understanding of their predictions. In contrast, in this work, we present an inverse perspective of distractor features: features that cast doubt about the prediction by affecting the model's confidence in its prediction. Understanding distractors provide a complementary view of the features' relevance in the predictions of neural models. In this paper, we apply a reduction-based technique to find distractors and provide our preliminary results of their impacts and types. Our experiments across various tasks, models, and datasets of code reveal that the removal of tokens can have a significant impact on the confidence of models in their predictions and the categories of tokens can also play a vital role in the model's confidence. Our study aims to enhance the transparency of models by emphasizing those tokens that significantly influence the confidence of the models.
{"title":"Study of Distractors in Neural Models of Code","authors":"Md Rafiqul Islam Rabin, Aftab Hussain, Sahil Suneja, Mohammad Amin Alipour","doi":"10.1109/InteNSE59150.2023.00005","DOIUrl":"https://doi.org/10.1109/InteNSE59150.2023.00005","url":null,"abstract":"Finding important features that contribute to the prediction of neural models is an active area of research in explainable AI. Neural models are opaque and finding such features sheds light on a better understanding of their predictions. In contrast, in this work, we present an inverse perspective of distractor features: features that cast doubt about the prediction by affecting the model's confidence in its prediction. Understanding distractors provide a complementary view of the features' relevance in the predictions of neural models. In this paper, we apply a reduction-based technique to find distractors and provide our preliminary results of their impacts and types. Our experiments across various tasks, models, and datasets of code reveal that the removal of tokens can have a significant impact on the confidence of models in their predictions and the categories of tokens can also play a vital role in the model's confidence. Our study aims to enhance the transparency of models by emphasizing those tokens that significantly influence the confidence of the models.","PeriodicalId":166762,"journal":{"name":"2023 IEEE/ACM International Workshop on Interpretability and Robustness in Neural Software Engineering (InteNSE)","volume":"258 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-03-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132323119","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：