Squall: Stream Processing and Analysis Model Design

Recently, the importance of velocity, one of the characteristics of big data (5V: Volume, Variety, Velocity, Veracity, and Value), has been emphasized in the data processing, which has led to several studies on the real-time stream processing, a technology for quick and accurate processing and analyses of big data. In this study, we propose a Squall framework using in-memory technology. Moreover, we provide a description of Squall framework and its operations. This Squall framework can support the real-time event stream processing and micro-batch processing, showing high performance and memory efficiency for stream processing using Go's excellent concurrency and GC (Garbage Collection) available without a virtual machine. Therefore, you can run many jobs on one machine. In addition, the data flows through the memory, the number of operation steps are incorporated to improve the performance. It provides relatively good performance compared to existing Apache Storm and spark streaming. In conclusion, it can be used as a general-purpose big data processing framework because it can overcome the drawbacks of existing Apache storm or Spark streaming by introducing the advantages of Go language.