# Zero Bubble Pipeline Parallelism. (arXiv:2401.10241v1 [cs.DC])


Pipeline parallelism is one of the key components for large-scale distributed
training, yet its efficiency suffers from pipeline bubbles which were deemed
inevitable. In this work, we introduce a scheduling strategy that, to our
knowledge, is the first to successfully achieve zero pipeline bubbles under
synchronous training semantics. The key idea behind this improvement is to split
the backward computation into two parts, one that computes gradient for the
input and another that computes for the parameters. Based on this idea, we
handcraft novel pipeline schedules that significantly outperform the baseline
methods. We further develop an algorithm that automatically finds

[Read Full Article](http://arxiv.org/abs/2401.10241)