By pipelining different sub-sequences of layers on separate accelerators, GPipe provides the flexibility of scaling a variety of different networks to gigantic sizes efficiently. To address the need for efficient and task-independent model parallelism, we introduce GPipe, a pipeline parallelism library that allows scaling any network that can be expressed as a sequence of layers. These solutions are often architecture-specific and do not transfer to other tasks. In many cases, increasing model capacity beyond the memory limit of a single accelerator has required developing special algorithms or infrastructure. Scaling up deep neural network capacity has been known as an effective approach to improving model quality for several different machine learning tasks.

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (Nov 2018, arXiv 2018) We conduct detailed analysis of the main components that lead to high transfer performance. On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). BiT performs well across a surprisingly wide range of data regimes - from 1 example per class to 1M total examples. By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision.

Benchmark ai