Data: Layer $f_i$ , input activation $z_{i-1}$

CPU pinned memory buffer $P_{i-1}$

CPU thread $T_{data}$

CUDA events $E_{data}^i$ , $E_{comp}^i$

CUDA Streams $S_{data}$ , $S_{comp}$

Result: $z_i$

Allocate (z_i);

$S_{comp}\Leftarrow z_i\leftarrow f_i\left(z_{i-1}\right)$ ;

$S_{comp}\Leftarrow E_{comp}^i$ ;

In Thread $T_{data}$ :

$S_{data}\Leftarrow P_{i-1}\leftarrow z_{i-1}$ ;

$S_{data}\Leftarrow E_{data}^i$ ;

Wait ( $E_{data}^i$ , $E_{comp}^i$ );

Free ( $z_{i-1}$ ).