Joint Optimization of Topology and Hyperparameters of Hybrid DNNs for Sentence Classification

Abstract

Deep Neural Networks (DNN) require specifically tuned architectures and hyperparameters when being applied to any given task. Nature-inspired algorithms have been successfully applied for optimising various hyperparameters in different types of DNNs such as convolutional and recurrent for sentence classification. Hybrid networks, which contain multiple types of neural architectures have more recently been used for sentence classification in order to achieve better performance. However, the inclusion of hybrid architectures creates numerous possibilities of designing the network and those sub-networks also need fine-tuning. At present these hybrid networks are designed manually and various organisation attempts are noticed. In order to understand the benefit and the best design principle of such hybrid DNNs for sentence classification, in this work we used an Evolutionary Algorithm (EA) to optimise the topology and various hyperparameters in different types of layers within the network. In our experiments, the proposed EA designed the hybrid networks by using a single dataset and evaluated the evolved networks on multiple other datasets to validate their generalisation capability. We compared the EA-designed hybrid networks with human-designed hybrid networks in addition to other EA-optimised and expert-designed non-hybrid architectures.