Data processing for large database using feature selection

Big data is a term for huge amount of data sets that are becoming more complex for data processing applications and are inadequate to deal with them. Big data need a set of techniques and technologies which can easily handle the complex data set and can be easily processed. Feature selection techniques have become an apparent need in many applications to identify the required information from a large set of data. Existing systems processes the same data repeatedly each time when a user request is submitted even for a small task. In this work, a feature selection technique using spark streaming is proposed which can get live stream of input data and process it in batches to extract the required feature from the given input stream data. The proposed method will use the feature selection technique to extract the required data from the large dataset based on the user requirements. The algorithm proposed will also help to increase the throughput of the system.