Enhanced sales forecasting model using textual search data: Fusing dynamics with big data

Forecasting sales is an essential marketing function, and, for most businesses, sales are driven by own and competitive activities. Most firms use their own marketing efforts or a selection of their competitor’s marketing efforts for forecasting sales. Due to data availability limitations, data on the full set of competitors are rarely used when forecasting sales. The emergence of online search data provides access to a novel data source on own as well as never-before observed competitive activities. We propose a novel regularized dynamic forecasting model utilizing all available competitive search data in a market vs. constructing ad-hoc and potentially subjective smaller competitive sets. Our model addresses the inherent statistical issue that arises when including a large number of competitive effects and parsimoniously utilizes all competitive data. We demonstrate our model using data from the US automobile industry over a twelve-year period and forecast car-model sales for 14 exemplary car-models utilizing multiple search measures for all 374 potential competitive car-models. We show that our model fits and forecasts sales better than models not leveraging the full competitive search data, e.g., using subjective sets of relevant competitors or narrowly defined category competitors. We also find that market research done via novel large-language models (also called LLMs) to obtain a narrower set of competitors does not outperform our proposed model that includes the full set of competitors.