Flow-Based Variational Sequence Autoencoder

Abstract

Posterior collapse, also known as the Kullback-Leibler (KL) vanishing, is a long-standing problem in variational recurrent autoencoder (VRAE) which is essentially developed for sequence generation. To alleviate the vanishing problem, a complicated latent variable is required instead of assuming it as standard Gaussian. Normalizing flow was proposed to build the bijective neural network which converts a simple distribution into a complex distribution. The resulting approximate posterior is closer to real posterior for better sequence generation. The KL divergence in learning objective is accordingly preserved to enrich the capability of generating the diverse sequences. This paper presents the flow-based VRAE to build the disentangled latent representation for sequence generation. KL preserving flows are exploited for conditional VRAE and evaluated for text representation as well as dialogue generation. In the im-plementation, the schemes of amortized regularization and skip connection are further imposed to strengthen the embedding and prediction. Experiments on different tasks show the merit of this latent variable representation for language modeling, sentiment classification and dialogue generation.