Algorithm: online actor-critic

For t = 1 to T − 1 do:

1) In s_t, take action a_t ~ π, get $\left(s_t,a_t,r_t,s_t{}_{+1}\right)$

2) update V with target $r_t+\gamma V\left(s_{t+1}\right)$

3) evaluate $\text{A}\left(s_t,a_t\right)=\text{target}-V\left(s_t\right)$

4) ${\nabla }_{\theta }{J}_{\theta }\approx {\nabla }_{\theta }\log {\pi }_{\theta }\left(a_t|s_t\right)\text{A}\left(s_t,a_t\right)$

5) $\theta \leftarrow \theta +a{\nabla }_{\theta }{J}_{\theta }$

end for

where α is the learning rate and γ is a discount factor.