When crossing a crowded area, a person can predict dangers or collisions in advance around him/her, and then makes a suitable decision which direction he/she should take. The pedestrian trajectory prediction aims at simulating such ability of humans in a crowded environment. Most of the existing trajectory predictions are all based on the traditional hand-crafted methods that often ignore critical factors and can only be adapted to specific environments. Based on deep learning technology, this paper proposes a data-driven pedestrian trajectory predictor called MLP-social-GRU. First, the proposed predictor processes a pedestrian trajectory with a Multilayer Perceptron (MLP). Then, it adopts Gated Recurrent Units (GRU) to get hidden features of a pedestrian motion patterns, from which relationships between pedestrians can be simulated. Next, the social-pooling is used to receive and merge the hidden status information to get the mutual influence of adjacent pedestrians. Finally, a unified pedestrian trajectory prediction framework is designed based on abovementioned modules. We evaluate our predictor on two publicly available datasets, ETH and UCY, and the results show that it is superior to popular models such as LSTM, social-LSTM, and goal-social-array.
{"title":"Pedestrian Trajectory Prediction with MLP-social-GRU","authors":"Yanbo Zhang, Liying Zheng","doi":"10.1145/3457682.3457737","DOIUrl":"https://doi.org/10.1145/3457682.3457737","url":null,"abstract":"When crossing a crowded area, a person can predict dangers or collisions in advance around him/her, and then makes a suitable decision which direction he/she should take. The pedestrian trajectory prediction aims at simulating such ability of humans in a crowded environment. Most of the existing trajectory predictions are all based on the traditional hand-crafted methods that often ignore critical factors and can only be adapted to specific environments. Based on deep learning technology, this paper proposes a data-driven pedestrian trajectory predictor called MLP-social-GRU. First, the proposed predictor processes a pedestrian trajectory with a Multilayer Perceptron (MLP). Then, it adopts Gated Recurrent Units (GRU) to get hidden features of a pedestrian motion patterns, from which relationships between pedestrians can be simulated. Next, the social-pooling is used to receive and merge the hidden status information to get the mutual influence of adjacent pedestrians. Finally, a unified pedestrian trajectory prediction framework is designed based on abovementioned modules. We evaluate our predictor on two publicly available datasets, ETH and UCY, and the results show that it is superior to popular models such as LSTM, social-LSTM, and goal-social-array.","PeriodicalId":142045,"journal":{"name":"2021 13th International Conference on Machine Learning and Computing","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-02-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132772021","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper mainly studies the implementation of diverse conversation generation system based on end-to-end neural networks and sentence classification on sentence function. In existing work, the output of the conversational system is mostly in the form of one type of sentence, for instance, declarative sentence. The type of sentence function is not well distributed. There is still insufficient diversity in the output of conversational system, which could be unattractive to users. A good conversational system could well interact with users by generating diverse output, including asking and responding, driving conversations to go further. Generating different type of output sentence is necessary to conversational systems, which is also challenging. Therefore, this paper introduces the idea of diverse conversation into the generative system, and designs the Diverse Conversation Generation (DCG) model. The model adopts a sentence function classifier trained independently to supervise the model output with modified loss function and back-propagation. The DCG model increases the diversity of output sentence, which could guide user to chat more with the system, extend the quality of conversation, and improve the user experience. The model is experimented on two different sequence-to sequence models, evaluated with perplexity and classify entropy, achieves better performance compared with two base models.
{"title":"Diverse Conversation Generation System with Sentence Function Classification","authors":"Zuning Fan, Liangwei Chen","doi":"10.1145/3457682.3457761","DOIUrl":"https://doi.org/10.1145/3457682.3457761","url":null,"abstract":"This paper mainly studies the implementation of diverse conversation generation system based on end-to-end neural networks and sentence classification on sentence function. In existing work, the output of the conversational system is mostly in the form of one type of sentence, for instance, declarative sentence. The type of sentence function is not well distributed. There is still insufficient diversity in the output of conversational system, which could be unattractive to users. A good conversational system could well interact with users by generating diverse output, including asking and responding, driving conversations to go further. Generating different type of output sentence is necessary to conversational systems, which is also challenging. Therefore, this paper introduces the idea of diverse conversation into the generative system, and designs the Diverse Conversation Generation (DCG) model. The model adopts a sentence function classifier trained independently to supervise the model output with modified loss function and back-propagation. The DCG model increases the diversity of output sentence, which could guide user to chat more with the system, extend the quality of conversation, and improve the user experience. The model is experimented on two different sequence-to sequence models, evaluated with perplexity and classify entropy, achieves better performance compared with two base models.","PeriodicalId":142045,"journal":{"name":"2021 13th International Conference on Machine Learning and Computing","volume":"70 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-02-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115716901","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Because it can provide real-time images for the first time, UAV plays a massive role in disaster relief, environmental observation, and information collection. However, the quality of images collected by UAV is always affected by fog. Therefore, the research on how to remove the fog in the image becomes more and more critical. In recent years, the role of convolutional neural networks (CNN), which can automatically extract features and efficiently process high-dimensional data, has received more and more attention in many disciplines. To improve the imaging quality of UAV in a foggy environment, this paper proposes an image dehazing model built with a convolutional neural network (CNN), called an effective end-to-end dehazing Network (AEE-Net). Our proposed method has a faster running speed than traditional models due to the simple structure of the model and the design based on the modified atmospheric scattering model. Our method combines the characteristics of dehazing processes and the advantages of deep learning. Experimental results on the training set and raw images show that the proposed method has better performance than traditional methods. This method can improve the quality of UAV-captured images under foggy conditions and can meet the input requirements of UAV vision tasks.
{"title":"AEE-Net: An Efficient End-to-End Dehazing Network in UAV Imaging System","authors":"Tianxiao Cai, Sheng Zhang, Bo Tan","doi":"10.1145/3457682.3457739","DOIUrl":"https://doi.org/10.1145/3457682.3457739","url":null,"abstract":"Because it can provide real-time images for the first time, UAV plays a massive role in disaster relief, environmental observation, and information collection. However, the quality of images collected by UAV is always affected by fog. Therefore, the research on how to remove the fog in the image becomes more and more critical. In recent years, the role of convolutional neural networks (CNN), which can automatically extract features and efficiently process high-dimensional data, has received more and more attention in many disciplines. To improve the imaging quality of UAV in a foggy environment, this paper proposes an image dehazing model built with a convolutional neural network (CNN), called an effective end-to-end dehazing Network (AEE-Net). Our proposed method has a faster running speed than traditional models due to the simple structure of the model and the design based on the modified atmospheric scattering model. Our method combines the characteristics of dehazing processes and the advantages of deep learning. Experimental results on the training set and raw images show that the proposed method has better performance than traditional methods. This method can improve the quality of UAV-captured images under foggy conditions and can meet the input requirements of UAV vision tasks.","PeriodicalId":142045,"journal":{"name":"2021 13th International Conference on Machine Learning and Computing","volume":"94 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-02-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116317446","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Penetration test is an important means to test the security of the web system. It has been mainly carried out by tester manually. The main reason is that it is difficult to generate test path and code automatically because of the complex network environment. The traditional method for attack path can't give the code for the whole penetration process. The traditional penetration path is based on the correlation between vulnerabilities and lacks practical experience support. In this paper, we propose a method based on CNN, which can automatically produce the code of penetration test by training the data which originate from the real attack events. We further implement the system to verify it. In a real environment experiment, we have validated the system, and analyzed the feasibility and performance of the CNN technology for penetration tests.
{"title":"Using Deep Learning to Construct Auto Web Penetration Test","authors":"Jian Jiao, Haini Zhao, Hongsheng Cao","doi":"10.1145/3457682.3457691","DOIUrl":"https://doi.org/10.1145/3457682.3457691","url":null,"abstract":"Penetration test is an important means to test the security of the web system. It has been mainly carried out by tester manually. The main reason is that it is difficult to generate test path and code automatically because of the complex network environment. The traditional method for attack path can't give the code for the whole penetration process. The traditional penetration path is based on the correlation between vulnerabilities and lacks practical experience support. In this paper, we propose a method based on CNN, which can automatically produce the code of penetration test by training the data which originate from the real attack events. We further implement the system to verify it. In a real environment experiment, we have validated the system, and analyzed the feasibility and performance of the CNN technology for penetration tests.","PeriodicalId":142045,"journal":{"name":"2021 13th International Conference on Machine Learning and Computing","volume":"271 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-02-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115997677","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The face forgery detection model based on XceptionNet has made great achievements in the detection field. However, it is still a challenge to detect fake faces in low-quality video images, because the low-quality video image has an insufficient resolution, which leads to the loss of details of the video image, thus resulting in image blurring. To solve this problem, this paper proposes a low-quality video face forgery detection method based on video recovery. This method mainly uses the Pyramid, Cascading, and Deformable convolution(PCD) module and the spatiotemporal attention (TSA) fusion module to restore low-quality video face images, and then obtains the restored feature map. And then the restored feature map is fed into the Xception classification network for face forgery detection. Moreover, The pre-training model parameters based on ImageNet makes the training model converge on 2GPU days. The results show that this method has a good experimental effect on the test data set.
基于XceptionNet的人脸伪造检测模型在检测领域取得了很大的成就。然而，在低质量视频图像中检测假脸仍然是一个挑战，因为低质量视频图像的分辨率不足，导致视频图像的细节丢失，从而导致图像模糊。为了解决这一问题，本文提出了一种基于视频恢复的低质量视频人脸伪造检测方法。该方法主要利用金字塔级联变形卷积(Pyramid, Cascading, and Deformable convolution, PCD)模块和时空注意力(spatial - temporal attention, TSA)融合模块对低质量视频人脸图像进行恢复，然后得到恢复后的特征图。然后将恢复的特征图输入异常分类网络进行人脸伪造检测。此外，基于ImageNet的预训练模型参数使训练模型在2GPU天内收敛。结果表明，该方法在测试数据集上具有良好的实验效果。
{"title":"Using Video Restoration to Improve Face Forgery Detection Based on Low-quality Video","authors":"Jianyang Qi, Peng Liang, Gang Hao, Yuting Wu","doi":"10.1145/3457682.3457753","DOIUrl":"https://doi.org/10.1145/3457682.3457753","url":null,"abstract":"The face forgery detection model based on XceptionNet has made great achievements in the detection field. However, it is still a challenge to detect fake faces in low-quality video images, because the low-quality video image has an insufficient resolution, which leads to the loss of details of the video image, thus resulting in image blurring. To solve this problem, this paper proposes a low-quality video face forgery detection method based on video recovery. This method mainly uses the Pyramid, Cascading, and Deformable convolution(PCD) module and the spatiotemporal attention (TSA) fusion module to restore low-quality video face images, and then obtains the restored feature map. And then the restored feature map is fed into the Xception classification network for face forgery detection. Moreover, The pre-training model parameters based on ImageNet makes the training model converge on 2GPU days. The results show that this method has a good experimental effect on the test data set.","PeriodicalId":142045,"journal":{"name":"2021 13th International Conference on Machine Learning and Computing","volume":"8 16 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-02-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129137716","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
COVID-19 pandemic has gravely affected our societies and economies with severe consequences. To contain the spread of the disease, most governments around the world authorized unprecedented measures, including Morocco, which has closed the borders and adopted full lockdown between March and June 2020. However, these measures have resulted in economic loss and have led to dramatic changes in how businesses act and consumers behave. The main focus of this study was to examine the impact of the full lockdown on Moroccan enterprises based on the COVID-19 Moroccan business survey carried out by the High Commission for Planning (HCP). A three-stage analysis method was employed. First, multiple correspondence analysis (MCA) was used to reduce the dimensionality of the categorical variables, and k-means clustering algorithm was used to cluster the data, then decision tree algorithm was performed in order to interpret each cluster and the maximum accuracy achieved is 84.45%. Compared with the decision tree algorithm, an artificial neural network (ANN) with stratified 10-fold cross-validation was applied to the dataset and has reached an accuracy of 83.4%. The simulation results confirm the effectiveness of the proposed techniques for analyzing survey data.
{"title":"Toward an Effective Analysis of COVID-19 Moroccan Business Survey Data using Machine Learning Techniques","authors":"Imane Lasri, Anouar Riadsolh, Mourad Elbelkacemi","doi":"10.1145/3457682.3457690","DOIUrl":"https://doi.org/10.1145/3457682.3457690","url":null,"abstract":"COVID-19 pandemic has gravely affected our societies and economies with severe consequences. To contain the spread of the disease, most governments around the world authorized unprecedented measures, including Morocco, which has closed the borders and adopted full lockdown between March and June 2020. However, these measures have resulted in economic loss and have led to dramatic changes in how businesses act and consumers behave. The main focus of this study was to examine the impact of the full lockdown on Moroccan enterprises based on the COVID-19 Moroccan business survey carried out by the High Commission for Planning (HCP). A three-stage analysis method was employed. First, multiple correspondence analysis (MCA) was used to reduce the dimensionality of the categorical variables, and k-means clustering algorithm was used to cluster the data, then decision tree algorithm was performed in order to interpret each cluster and the maximum accuracy achieved is 84.45%. Compared with the decision tree algorithm, an artificial neural network (ANN) with stratified 10-fold cross-validation was applied to the dataset and has reached an accuracy of 83.4%. The simulation results confirm the effectiveness of the proposed techniques for analyzing survey data.","PeriodicalId":142045,"journal":{"name":"2021 13th International Conference on Machine Learning and Computing","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-02-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130571348","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Inter-domain link inference is not only important for network security and fault diagnosis, but also helps to conduct research on inter-domain congestion detection and network resilience assessment. Current researches on this issue lack confidence analysis of the inferred results. In this paper, the IP link types (i.e., intra-domain link and inter-domain link) are considered as the latent variable in probability model, while the parameters are probabilities of different link types with particular features. The expectation maximization algorithm is applied to estimate parameters of the model. In each iteration of EM algorithm, Naïve Bayes is used for classification. The final result is determined according to the probability, and the probability is the confidence of the result. The experimental results show that our method can achieve better precision and recall on the validation set than two existing general methods.
{"title":"Inter-domain Link Inference with Confidence Using Naïve Bayes Classifier","authors":"Yi Zhao, Yan Liu, Xiaoyu Guo, ZhongHang Sui","doi":"10.1145/3457682.3457715","DOIUrl":"https://doi.org/10.1145/3457682.3457715","url":null,"abstract":"Inter-domain link inference is not only important for network security and fault diagnosis, but also helps to conduct research on inter-domain congestion detection and network resilience assessment. Current researches on this issue lack confidence analysis of the inferred results. In this paper, the IP link types (i.e., intra-domain link and inter-domain link) are considered as the latent variable in probability model, while the parameters are probabilities of different link types with particular features. The expectation maximization algorithm is applied to estimate parameters of the model. In each iteration of EM algorithm, Naïve Bayes is used for classification. The final result is determined according to the probability, and the probability is the confidence of the result. The experimental results show that our method can achieve better precision and recall on the validation set than two existing general methods.","PeriodicalId":142045,"journal":{"name":"2021 13th International Conference on Machine Learning and Computing","volume":"10 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-02-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117013761","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Intensive information is emerged in social media every second. Many breaking news often appear first in social media, much earlier than they appear in traditional news media. Through the technology of event detection on social media data streams, scatter information can be gathered together to inform us the popular events discussing online. An event is often modeled as a cluster of documents which discuss the same subject. Traditional event detection methods perform poorly on social media because of their huge amount of data and irregular expressions. In this paper, we propose a simple yet efficient event detection method towards social media. An event is represented by a sequence of keywords extracted from social media. We use a single-pass incremental clustering method with a trained encoder mapping documents and events into the same semantic space, which is helpful for the similarity calculation between them. We consider the similarity calculation between a tweet and an event as a matching process and construct a relevance matching dataset with tweet-event pairs. We finetune BERT (Bidirectional Encoder Representations from Transformers) model in the matching dataset to get an appropriate semantic encoder. Keywords are dynamically changed to represent an event for capturing the development of the event. Our proposed method achieves 0.86 on NMI (Normed Mutual Information), 0.69 on ARI (Adjusted Rand Index) and 0.70 on F1-score on a public twitter dataset, which shows the superiority of our method compared with baseline methods.
{"title":"Single-Pass On-Line Event Detection in Twitter Streams","authors":"Xingfa Qiu, Qiaosha Zou, C. Richard Shi","doi":"10.1145/3457682.3457762","DOIUrl":"https://doi.org/10.1145/3457682.3457762","url":null,"abstract":"Intensive information is emerged in social media every second. Many breaking news often appear first in social media, much earlier than they appear in traditional news media. Through the technology of event detection on social media data streams, scatter information can be gathered together to inform us the popular events discussing online. An event is often modeled as a cluster of documents which discuss the same subject. Traditional event detection methods perform poorly on social media because of their huge amount of data and irregular expressions. In this paper, we propose a simple yet efficient event detection method towards social media. An event is represented by a sequence of keywords extracted from social media. We use a single-pass incremental clustering method with a trained encoder mapping documents and events into the same semantic space, which is helpful for the similarity calculation between them. We consider the similarity calculation between a tweet and an event as a matching process and construct a relevance matching dataset with tweet-event pairs. We finetune BERT (Bidirectional Encoder Representations from Transformers) model in the matching dataset to get an appropriate semantic encoder. Keywords are dynamically changed to represent an event for capturing the development of the event. Our proposed method achieves 0.86 on NMI (Normed Mutual Information), 0.69 on ARI (Adjusted Rand Index) and 0.70 on F1-score on a public twitter dataset, which shows the superiority of our method compared with baseline methods.","PeriodicalId":142045,"journal":{"name":"2021 13th International Conference on Machine Learning and Computing","volume":"88 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-02-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127207776","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
With the open-source revolution, source codes are now more easily accessible than ever. This has, however, made it easier for malicious users and institutions to copy the code without giving regards to the license, or credit to the original author. Therefore, source code author identification is a critical task with paramount importance. In this paper, we propose ICodeNet - a hierarchical neural network that can be used for source code file-level tasks. The ICodeNet processes source code in image format and is employed for the task of per file author identification. The ICodeNet consists of an ImageNet trained VGG encoder followed by a shallow neural network. The shallow network is based either on CNN or LSTM. Different variations of models are evaluated on a source code author classification dataset. We have also compared our image-based hierarchical neural network model with simple image-based CNN architecture and text-based CNN and LSTM models to highlight its novelty and efficiency.
{"title":"ICodeNet - A Hierarchical Neural Network Approach For Source Code Author Identification","authors":"Pranali Bora, Tulika Awalgaonkar, Himanshu Palve, Raviraj Joshi, Purvi Goel","doi":"10.1145/3457682.3457709","DOIUrl":"https://doi.org/10.1145/3457682.3457709","url":null,"abstract":"With the open-source revolution, source codes are now more easily accessible than ever. This has, however, made it easier for malicious users and institutions to copy the code without giving regards to the license, or credit to the original author. Therefore, source code author identification is a critical task with paramount importance. In this paper, we propose ICodeNet - a hierarchical neural network that can be used for source code file-level tasks. The ICodeNet processes source code in image format and is employed for the task of per file author identification. The ICodeNet consists of an ImageNet trained VGG encoder followed by a shallow neural network. The shallow network is based either on CNN or LSTM. Different variations of models are evaluated on a source code author classification dataset. We have also compared our image-based hierarchical neural network model with simple image-based CNN architecture and text-based CNN and LSTM models to highlight its novelty and efficiency.","PeriodicalId":142045,"journal":{"name":"2021 13th International Conference on Machine Learning and Computing","volume":"69 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-01-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126228653","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Francesca Del Bonifro, M. Gabbrielli, Stefano Zacchiroli
Programming language detection is a common need in the analysis of large source code bases. It is supported by a number of existing tools that rely on several features, and most notably file extensions, to determine file types. We consider the problem of accurately detecting the type of files commonly found in software code bases, based solely on textual file content. Doing so is helpful to classify source code that lack file extensions (e.g., code snippets posted on the Web or executable scripts), to avoid misclassifying source code that has been recorded with wrong or uncommon file extensions, and also shed some light on the intrinsic recognizability of source code files. We propose a simple model that (a) use a language-agnostic word tokenizer for textual files, (b) group tokens in 1-/2-grams, (c) build feature vectors based on N-gram frequencies, and (d) use a simple fully connected neural network as classifier. As training set we use textual files extracted from GitHub repositories with at least 1000 stars, using existing file extensions as ground truth. Despite its simplicity the proposed model reaches ≈ 85% in our experiments for a relatively high number of recognized classes (more than 130 file types).
{"title":"Content-Based Textual File Type Detection at Scale","authors":"Francesca Del Bonifro, M. Gabbrielli, Stefano Zacchiroli","doi":"10.1145/3457682.3457756","DOIUrl":"https://doi.org/10.1145/3457682.3457756","url":null,"abstract":"Programming language detection is a common need in the analysis of large source code bases. It is supported by a number of existing tools that rely on several features, and most notably file extensions, to determine file types. We consider the problem of accurately detecting the type of files commonly found in software code bases, based solely on textual file content. Doing so is helpful to classify source code that lack file extensions (e.g., code snippets posted on the Web or executable scripts), to avoid misclassifying source code that has been recorded with wrong or uncommon file extensions, and also shed some light on the intrinsic recognizability of source code files. We propose a simple model that (a) use a language-agnostic word tokenizer for textual files, (b) group tokens in 1-/2-grams, (c) build feature vectors based on N-gram frequencies, and (d) use a simple fully connected neural network as classifier. As training set we use textual files extracted from GitHub repositories with at least 1000 stars, using existing file extensions as ground truth. Despite its simplicity the proposed model reaches ≈ 85% in our experiments for a relatively high number of recognized classes (more than 130 file types).","PeriodicalId":142045,"journal":{"name":"2021 13th International Conference on Machine Learning and Computing","volume":"52 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-01-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116766747","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：