{"title":"Correlation and Visualization Analysis of Large Scale Dataset GDELT","authors":"Fengcai Qiao, Kedi Chen","doi":"10.1109/IIKI.2016.19","DOIUrl":null,"url":null,"abstract":"The Global Data on Events, Location, and Tone (GDELT) is a real time large scale database of global human society for open research which monitors the worlds broadcast, print, and web news since 1979, creating a free open platform for computing on the entire world. In this paper, first, we designed and implemented a data crawler, which collects metadata of GDELT database in real time and stores them in Hadoop Distributed File System (HDFS). Then, we proposed a hashbased method to correlate \"Event\" table, \"Mentions\" table and \"GKG\" table in GDELT, in order to digest every detailed information of each event. Finally, we took South Korea as example to make spatiotemporal visualization analysis, such as event spatiotemporal heat map, distribution of media attention and event extraction confidence dot map.","PeriodicalId":371106,"journal":{"name":"2016 International Conference on Identification, Information and Knowledge in the Internet of Things (IIKI)","volume":"123 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2016-10-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"3","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2016 International Conference on Identification, Information and Knowledge in the Internet of Things (IIKI)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/IIKI.2016.19","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 3
Abstract
The Global Data on Events, Location, and Tone (GDELT) is a real time large scale database of global human society for open research which monitors the worlds broadcast, print, and web news since 1979, creating a free open platform for computing on the entire world. In this paper, first, we designed and implemented a data crawler, which collects metadata of GDELT database in real time and stores them in Hadoop Distributed File System (HDFS). Then, we proposed a hashbased method to correlate "Event" table, "Mentions" table and "GKG" table in GDELT, in order to digest every detailed information of each event. Finally, we took South Korea as example to make spatiotemporal visualization analysis, such as event spatiotemporal heat map, distribution of media attention and event extraction confidence dot map.
Global Data on Events, Location, and Tone (GDELT)是一个用于开放研究的全球人类社会的实时大型数据库，它监测了自1979年以来世界广播，印刷和网络新闻，为整个世界创造了一个免费开放的计算平台。本文首先设计并实现了一个数据爬虫，实时收集GDELT数据库的元数据并存储在Hadoop分布式文件系统(HDFS)中。然后，我们提出了一种基于哈希的方法来关联GDELT中的“Event”表、“mention”表和“GKG”表，以消化每个事件的每个详细信息。最后，以韩国为例，进行事件时空热图、媒体关注度分布、事件提取置信度点图等时空可视化分析。
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
