Data: Layer $f_i$ , output gradients $\frac{\delta \mathcal{L}}{\delta z_i}$

CPU pinned memory buffer $P_{i-1}$

CPU thread $T_{comp}$

CUDA events $E_{data}^i$ , $E_{data}^{i+1}$ , $E_{comp}^i$

CUDA Streams $S_{data}$ , $S_{comp}$

Result: $\frac{\delta \mathcal{L}}{\delta z_{i-1}}$ , $\frac{\delta \mathcal{L}}{\delta {\theta }_i}$

Allocate ( $z_{i-1}$ );

$S_{data}\Leftarrow z_{i-1}\leftarrow P_{i-1}$ ;

$S_{data}\Leftarrow E_{data}^i$ ;

Wait ( $E_{data}^{i+1}$ );

Allocate ( $\frac{\delta \mathcal{L}}{\delta z_{i-1}}$ , $\frac{\delta \mathcal{L}}{\delta {\theta }_i}$ );

$S_{comp}\Leftarrow \frac{\delta \mathcal{L}}{\delta z_{i-1}}\leftarrow \frac{\delta \mathcal{L}}{\delta z_i}\times \frac{\delta z_i}{\delta z_{i-1}}$ ;

$S_{comp}\Leftarrow \frac{\delta \mathcal{L}}{\delta {\theta }_i}\leftarrow \frac{\delta \mathcal{L}}{\delta z_i}\times \frac{\delta z_i}{\delta {\theta }_i}$ .