{"title":"情感分析系统中Tweets预处理和词干提取的高效算法","authors":"H. Al-Khafaji, A. Habeeb","doi":"10.9790/0661-1903024450","DOIUrl":null,"url":null,"abstract":"The preprocessing step approximately consumes 85% of the time and efforts of overall time and efforts of the Knowledge Discovery in Database, (KDD). Sentiments analysis, as a new trend in KDD and data mining, requires many preprocessing steps such as tokenization, stop words removing, and stemming. These steps play, in addition to their preparation role, the data reduction role by excluding worthless data and preserving significant data. This paper presents the design and implementation of a system for English tweets segmentation, cleaning, stop words removing, and stemming. This system implemented as MS-SQL Server stored procedures to be part of a tightly coupled sentiments mining system. Many experiments accomplished to prove the validity and efficiency of the system using different sizes data sets arranged from 250000 to 1000000 tweets and it accomplished the data reduction process to achieve considerable size reduction with preservation of significant data set's attributes. The system exhibited linear behavior according to the data size growth.","PeriodicalId":91890,"journal":{"name":"IOSR journal of computer engineering","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2017-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"14","resultStr":"{\"title\":\"Efficient Algorithms for Preprocessing and Stemming of Tweets in a Sentiment Analysis System\",\"authors\":\"H. Al-Khafaji, A. Habeeb\",\"doi\":\"10.9790/0661-1903024450\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The preprocessing step approximately consumes 85% of the time and efforts of overall time and efforts of the Knowledge Discovery in Database, (KDD). Sentiments analysis, as a new trend in KDD and data mining, requires many preprocessing steps such as tokenization, stop words removing, and stemming. These steps play, in addition to their preparation role, the data reduction role by excluding worthless data and preserving significant data. This paper presents the design and implementation of a system for English tweets segmentation, cleaning, stop words removing, and stemming. This system implemented as MS-SQL Server stored procedures to be part of a tightly coupled sentiments mining system. Many experiments accomplished to prove the validity and efficiency of the system using different sizes data sets arranged from 250000 to 1000000 tweets and it accomplished the data reduction process to achieve considerable size reduction with preservation of significant data set's attributes. The system exhibited linear behavior according to the data size growth.\",\"PeriodicalId\":91890,\"journal\":{\"name\":\"IOSR journal of computer engineering\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2017-05-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"14\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IOSR journal of computer engineering\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.9790/0661-1903024450\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IOSR journal of computer engineering","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.9790/0661-1903024450","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 14
摘要
预处理步骤大约消耗了KDD (Knowledge Discovery in Database)总体时间和精力的85%。情感分析作为KDD和数据挖掘的一个新趋势,需要许多预处理步骤,如标记化、停止词删除和词干提取。这些步骤除了发挥准备作用外,还通过排除无用数据和保留重要数据来减少数据。本文设计并实现了一个英语tweets的分词、清理、停词删除和词干提取系统。本系统实现为MS-SQL Server存储过程,作为一个紧密耦合的情感挖掘系统的一部分。使用25万到100万tweets的不同大小的数据集进行了大量实验,证明了系统的有效性和效率,并完成了数据约简过程,在保留重要数据集属性的情况下实现了相当大的规模缩减。随着数据量的增长,系统表现出线性行为。
Efficient Algorithms for Preprocessing and Stemming of Tweets in a Sentiment Analysis System
The preprocessing step approximately consumes 85% of the time and efforts of overall time and efforts of the Knowledge Discovery in Database, (KDD). Sentiments analysis, as a new trend in KDD and data mining, requires many preprocessing steps such as tokenization, stop words removing, and stemming. These steps play, in addition to their preparation role, the data reduction role by excluding worthless data and preserving significant data. This paper presents the design and implementation of a system for English tweets segmentation, cleaning, stop words removing, and stemming. This system implemented as MS-SQL Server stored procedures to be part of a tightly coupled sentiments mining system. Many experiments accomplished to prove the validity and efficiency of the system using different sizes data sets arranged from 250000 to 1000000 tweets and it accomplished the data reduction process to achieve considerable size reduction with preservation of significant data set's attributes. The system exhibited linear behavior according to the data size growth.