Implementing Recurrent Neural Networks in Process Systems Engineering applications, the right way!

Abstract

This manuscript identifies, addresses and illustrates via comparisons an inconsistency and inaccuracy with the implementation of Recurrent Neural Networks (RNNs) on naturally occurring dynamical processes, particularly focusing on implementations that claim to identify input–output dynamic relationships through a state–space representation. While the RNN structure does lend itself to these types of problems, there are two major issues with how RNNs are typically structured and trained in this context. Firstly, the hidden states of the model are commonly reinitialized or discarded between each of the input–output sequences in the training data set, essentially leading to a framework where the initial state for each sequence is not trained. In contrast, in a typical state–space model identification framework, the model parameters along with the states are (and need to be) identified together. Secondly, the model structure of the RNN is different from a classic state space (SS) representation. While in state space representations the current state is defined to be a function of the state and input from the previous time step, RNNs use input from the same time step. In this paper, two changes are proposed to address these inconsistencies. The first step is to train the initial hidden states for the training sequences. To address the structural inconsistency between a state space model and the RNN, the list of hidden states retrieved from the RNN is formatted to represent the data and state pairings that a state space model would create. The effect of these corrections is demonstrated in the simplest of dynamical systems — data generated using a Linear Time-Invariant (LTI) state space model. The importance of both these corrections is demonstrated by implementing them one at a time. Interestingly, the model that performed the worst in testing was the model with only the trained hidden states. The model with no changes was slightly better, and the model with the correct input timing but no trained hidden states increased performance by a significant amount. Finally, the best results were found when both changes were implemented.