A new pipeline with ultimate search efficiency for neural architecture search

Abstract

We present a novel neural architecture search pipeline designed to enhance search efficiency through optimized data and algorithms. Leveraging dataset distillation techniques, our pipeline condenses large-scale target datasets into more streamlined proxy datasets, effectively reducing the computational overhead associated with identifying optimal neural architectures. To accommodate diverse approaches to synthetic dataset utilization, our pipeline comprises two distinct schemes. Scheme 1 involves constructing rich data from various Bases $\left|B\right|$, while Scheme 2 focuses on establishing high-quality relationship mappings within the data. Models generated through Scheme 1 exhibit outstanding scalability, demonstrating superior performance when transferred to larger, more complex tasks. Despite utilizing fewer data, Scheme 2 maintains performance levels without degradation on the source dataset. Furthermore, our research extends to the inherent challenges present in DARTS-derived algorithms, particularly in the selection of candidate operations based on architectural parameters. We identify architectural parameter disparities across different edges, highlighting the occurrence of “Selection Errors” during the model generation process, and propose an enhanced search algorithm. Our proposed algorithm comprises three components—attention, regularization, and normalization—aiding in the rapid identification of high-quality models using data generated from proxy datasets. Experimental results demonstrate a significant reduction in search time, with high-quality models generated in as little as two minutes using our proposed pipeline. Through comprehensive experimentation, we meticulously validate the efficacy of both schemes and algorithms.