HDQN-based Algorithm for routing

Input: status of the node,

Output: routing policy

Initialize evaluation, target networks with parameters δ;

Initialize experience replay memory D;

for Episode = 1, 2, ..., N^eps do

Initialize state s_t;

for TS t = 1, 2, ...,T do

Obtain s_t;

Select $a_t=\arg \max Q\left(s_t,a_t\right)$ with probability $\epsilon$ ;

Randomly select a_t with probability $1-\epsilon$ ;

Forward the data to the next node, obtain the corresponding reward from formula and s_t+1;

Update the current state to the next latest state to get new network input;

Store transition {s_t, a_t, r_t, s_t+1} into experience replay memory;

if the learning process starts then

Randomly sample M transitions from experience replay memory;

Update evaluation network from formula;

Calculate the target Q-value for the current state: $y_i=\{\begin{array}{l}{r}_j,\mathrm{if}dataissuccessfullyforwardedto\mathrm{BS}\\ r_j+\gamma {\max }_{a_{t+1}}\widehat{Q}\left({\varphi }_{j+1},a_{t+1};{\theta }^{-}\right),\mathrm{otherwise}\end{array}$

Update target network periodically;

end if

end for

end for