NetAdaptV1

NetAdaptV2





NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization



Abstract

Neural architecture search (NAS) typically consists of three main steps: training a super-network, training and evaluating sampled deep neural networks (DNNs), and training the discovered DNN. Most of the existing efforts speed up some steps at the cost of a significant slowdown of other steps or sacrificing the support of non-differentiable search metrics. The unbalanced reduction in the time spent per step limits the total search time reduction, and the inability to support non-differentiable search metrics limits the performance of discovered DNNs.

In this paper, we present NetAdaptV2 with three innovations to better balance the time spent for each step while supporting non-differentiable search metrics. First, we propose channel-level bypass connections that merge network depth and layer width into a single search dimension to reduce the time for training and evaluating sampled DNNs. Second, ordered dropout is proposed to train multiple DNNs in a single forward-backward pass to decrease the time for training a super-network. Third, we propose the multi-layer coordinate descent optimizer that considers the interplay of multiple layers in each iteration of optimization to improve the performance of discovered DNNs while supporting non-differentiable search metrics. With these innovations, NetAdaptV2 reduces the total search time by up to 5.8x on ImageNet and 2.4x on NYU Depth V2, respectively, and discovers DNNs with better accuracy-latency/accuracy-MAC trade-offs than state-of-the-art NAS works. Moreover, the discovered DNN outperforms NAS-discovered MobileNetV3 by 1.8% higher top-1 accuracy with the same latency.


Algorithm Flow of NetAdaptV2

Algorithm Flow


Performance on ImageNet Validation Set (Mobile CPU on Google Pixel 1)

Performance on ImageNet Validation Set (Mobile CPU on Google Pixel 1)



Downloads


BibTeX


@InProceedings{cvpr_2021_yang_netadaptv2,
	author = {Yang, Tien-Ju and Liao, Yi-Lun and Sze, Vivienne},
	title = {NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization},
	booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
	month = {June},
	year = {2021}
}
                

NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications



Abstract

This work proposes an algorithm, called NetAdapt, that automatically adapts a pre-trained deep neural network to a mobile platform given a resource budget. While many existing algorithms simplify networks based on the number of MACs or weights, optimizing those indirect metrics may not necessarily reduce the direct metrics, such as latency and energy consumption. To solve this problem, NetAdapt incorporates direct metrics into its adaptation algorithm. These direct metrics are evaluated using empirical measurements, so that detailed knowledge of the platform and toolchain is not required. NetAdapt automatically and progressively simplifies a pre-trained network until the resource budget is met while maximizing the accuracy. Experiment results show that NetAdapt achieves better accuracy versus latency trade-offs on both mobile CPU and mobile GPU, compared with the state-of-the-art automated network simplification algorithms. For image classification on the ImageNet dataset, NetAdapt achieves up to a 1.7x speedup in measured inference latency with equal or higher accuracy on MobileNets (V1&V2).


Illustration of NetAdaptV1
Algorithm Flow of NetAdaptV1

Illustration of NetAdapt

Algorithm Flow of NetAdapt


Lookup Tables for Fast Resource Estimation
Performance on ImageNet Validation Set (MobileNet V1, Mobile CPU on Google Pixel 1)

Lookup Tables for Fast Resource Estimation

Performance on ImageNet Validation Set (MobileNet V1, Mobile CPU on Google Pixel 1)



Downloads


BibTeX


@InProceedings{eccv_2018_yang_netadapt,
	author = {Yang, Tien-Ju and Howard, Andrew and Chen, Bo and Zhang, Xiao and Go, Alec and Sandler, Mark and Sze, Vivienne and Adam, Hartwig},
	title = {NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications},
	booktitle = {The European Conference on Computer Vision (ECCV)},
	month = {September},
	year = {2018}
}