Penalized composite likelihood estimation for hidden Markov models with unknown number of states

Estimating hidden Markov models (HMMs) with unknown number of states is a challenging task. In this paper, we propose a new penalized composite likelihood approach for simultaneously estimating both the number of states and the parameters in an overfitted HMM. We prove the order selection consistency and asymptotic normality of the resultant estimator. Simulation studies and an application demonstrate the finite sample performance of the proposed method.