Stock market forecasting based on machine learning: The role of investor sentiment

Abstract

Stock market prediction remains a classical yet challenging problem, with the focus on the investor sentiment growing increasing significant in big data era. This analysis delves into the question whether and how predicable is the stock market when considering investor sentiment. By leveraging the initial and customized LM financial lexicon and Vader technology, Word2vec and Doc2vec and BERT embedding vector method (along with two fine-tuned models: FinBERT and SentiBERT), we first construct nine investor sentiment indexes based on the textual data from Twitter between November 2019 and December 2021. And then we employ three machine learning algorithms (SVR, AdaBoost, and RF) to predict the daily return of the S&P 500 index. The experiment results confirm that the investor sentiment index can enhance prediction accuracy beyond the market indicator, aligning with prior research. Embedding vector methods exhibit superior performance compared to the fine-tuned models, and the customized dictionaries outperform their traditional counterparts. Furthermore, the composite sentiment index, integrating all the single indexes, achieves the best overall performance. To further validate our findings, we conduct robustness checks on the DJIA index and across different economic cycles, observe that the single sentiment index performs worse with shorter datasets, whereas the composite index demonstrates consistent improvement in both volatile and steady periods. These findings offer valuable insights for future research and provide practical applications in stock market prediction.