Globally Assisted Instance Normalization for Bandwidth-Efficient Neural Style Transfer

Abstract

Instance normalization (IN) has been widely considered as a key technique in fast neural style transfer algorithms to generate high-quality stylized images. However, because of the calculations of channel-wise means and standard deviations, instance normalization requires layer-by-layer inference flow for CNN accelerators. This kind of dataflow results in huge DRAM bandwidth which is unaffordable for mobile devices or embedding applications. We propose a novel normalization method named globally assisted instance normalization (GAIN) which receives generated statistics from a global branch without actually calculating channel-wise means and standard deviations. Our method generates comparable stylized results and incorporates block-based inference flows to avoid intermediate data transmission. For fast neural style transfer at Full HD 30 fps and 4K UHD 60 fps, we only need 2.52 GB/s and 15.40 GB/s of DRAM bandwidth respectively, which are 90.22% and 92.53% lower than IN with the layer-by-layer inference flow method.