Research and implementation of efficient retrieval algorithm in big data environment

Abstract

Under the background of digital information age, faced with the increasing data scale and complexity, the application limitations of traditional centralized retrieval services are becoming more and more obvious, and it is urgent to improve the data structure expansion, incremental update control and retrieval operation efficiency. In this paper, the efficient retrieval algorithm and technology of massive data information are taken as the research object, and a set of construction scheme of big data storage and retrieval system is proposed for unstructured data, which promotes the organic combination of distributed technology and full-text retrieval technology and realizes the optimization of fast retrieval processing mode of large-scale data. The system is based on Hadoop framework, with Hbase as the data storage module, and combined with ElasticSearch engine, IKAnalyzer word breaker and Redis cache to complete real-time and efficient data retrieval. Finally, based on Java web technology, a network application program convenient for users to operate online is formed. Practice has proved that the system has solved many problems in the process of collecting, storing and retrieving massive unstructured text data. At the same time, it improves the sharing transmission efficiency and concurrent access control ability of data information, and opens up a brand-new big data retrieval service model.