Algorithm 1 Quantile Regression Q Learning [5]

Require: N, κ

Input $s,a,r,s^{\prime },$ $\gamma ϵ\left[0,1\right)$

#Compute distributional Bellman target

$Q\left(s^{\prime },a^{\prime }\right):={\displaystyle {\sum }_j^{}{q}_j{\theta }_j\left(s^{\prime },a^{\prime }\right)}$ where $q_j=1/N$

$a^{*}\leftarrow argma{x}_{a^{\prime }}Q\left(s^{\prime },a^{\prime }\right)$

$T{\theta }_j\leftarrow r+\gamma {\theta }_j\left(s^{\prime },a^{*}\right),\forall j$

#Compute quantile regression loss

Output ${\displaystyle {\sum }_{i=1}^N{E}_j}\left[{\rho }_{\widehat{\tau }}^k{}_{{}_i}\left(T{\theta }_j-Q\left(s,a\right)\right)\right]$