Real-Time Weather Analytics: An End-to-End Big Data Analytics Service Over Apach Spark With Kafka and Long Short-Term Memory Networks

Abstract

Weather forecasting is one of the biggest challenges that modern science is still contending with. The advent of high-power computing, technical advancement of data storage devices, and incumbent reduction in the storage cost have accelerated data collection to turmoil. In this background, many artificial intelligence techniques have been developed and opened interesting window of opportunity in hitherto difficult areas. India is on the cusp of a major technology overhaul with millions of people's data availability who were earlier unconnected with the internet. The country needs to fast forward the innovative use of available data. The proposed model endeavors to forecast temperature, precipitation, and other vital information for usability in the agrarian sector. This project intends to develop a robust weather forecast model that learns automatically from the daily feed of weather data that is input through a third-party API source. The weather feed is sourced from openweathermap, an online service that provides weather data, and is streamed into the forecast model through Kafka components. The LSTM neural network used by the forecast model is designed to continuously learn from predictions and perform actual analysis. The model can be architected to be implemented across very large applications having the capability to process large volumes of streamed or stored data.