The explosion of data volume and the expansion in data dimensionality have led to a critical challenge in analyzing high-dimensional matrix time series for big data-related applications. In this regard, factor models for matrix-valued high-dimensional time series provide a powerful tool, that reduces the dimensionality of the variables with low-rank structures. However, existing high-dimensional matrix factor models suffer from two limitations in complex scenarios. One is that it is difficult to make robust inferences for datasets with heavy-tailed distributions. The other is that existing models require additional parameters for fine-tuning to guarantee performance. We propose an adaptively robust high-dimensional matrix factor model based on a specified Huber loss function to tackle the challenges mentioned above. An efficient iterative algorithm is provided to consistently determine the additional parameters of our model for robust estimation. The robustness of the model estimation is greatly improved by incorporating the Huber loss. Furthermore, we theoretically investigate the proposed method and derive the convergence rates of the robust estimators to examine its performance. Simulations show that the proposed method outperforms previous models in the estimation of heavy-tailed data. A real-world data analysis on a financial portfolio dataset illustrates that the method can be used to extract useful knowledge from high-dimensional matrix time series.