Double DQN

This page was last edited on 15 May 2025

Double DQN is an improved version of the original DQN (Deep Q-Network). It fixes overestimation bias by decoupling action selection from action evaluation.

Why Choose Double DQN?

DQN often overestimates Q-values, which leads to suboptimal policies.
Double DQN reduces this bias → more stable learning, better generalization.

Same architecture, better behavior.

What Type of Learning Is It?

Double DQN is a value-based, deep reinforcement learning algorithm.
It uses a neural network to estimate Q-values for each action.

Model-Free or Model-Based?

Model-Free.
It doesn’t learn the environment dynamics.
It learns from sampled experience only.

What Does It Try to Optimize?

It tries to minimize the TD error (temporal difference).
The target is based on the greedy action from the main network, but the value from the target network.

Training Loop (High-Level)

1. Collect experience: (state, action, reward, next_state, done)
2. Store in replay buffer
3. Sample a minibatch
4. Choose action from main_net → argmax_a Q(s', a)
5. Evaluate action using target_net → Q(s', a_main)
6. Compute target: target = r + γ * Q_target(s', a_main)
7. Compute loss: MSE between predicted and target Q
8. Backpropagate loss and update main_net
9. Soft update or periodic copy of weights to target_net

How to Implement Double DQN?

Use same structure as DQN, with this difference in the target:

a_next = argmax(main_net(next_state))
target_Q = target_net(next_state)[a_next]
target = reward + gamma * target_Q

On-Policy or Off-Policy?

Off-Policy.
Agent learns from past transitions collected by an older policy.
Experience replay makes this possible.

Exploration vs. Exploitation?

Uses ε-greedy policy:

• With probability ε → pick random action (explore)
• Else → pick best action from Q-values (exploit)

ε decays over time to shift focus from exploration to exploitation.

When Does It Converge?

Converges under same conditions as DQN:

• Sufficient replay buffer diversity
• Proper learning rate
• Slow updates of target network
• Enough exploration

Double DQN is more stable and converges better than DQN in noisy environments.

Where Does It Struggle?

• In continuous action spaces (you need DDPG, SAC, etc.)
• If Q-network architecture is poor
• If target network is not updated correctly
• With too small or non-diverse replay buffer

What Problems Is It Good For?

• Discrete action spaces
• Atari games, simple robotics, control problems
• When overestimation harms performance
• Long episodes with sparse rewards

Common Traps and Mistakes

• Using target = max(target_net) → That’s DQN, not Double DQN
• Updating target network too often or too fast
• Too small replay buffer
• Noisy reward functions without regularization
• Using too large ε at the end of training

Double DQN Equation

Original DQN Target:

   

Double DQN Target:

   

Where:

• y – the target Q-value that we want to approximate
• r – the reward received after taking action a in state s
• γ – the discount factor (0 < γ ≤ 1), controls how much we care about future rewards
• s' – the next state after taking action a
• Q_main(s', a) – the main (online) network, used to select the best action
• argmax_a Q_main(s', a) – selects the action with the highest Q-value in the next state, using the main network
• Q_target(s', ...) – evaluates the value of that action using the target network
• Q_target(s', argmax_a Q_main(s', a)) – the Double DQN target: value from target network, action from main network

What changed?

We decouple action selection (from main Q) and action evaluation (from target Q). This prevents overestimation.

Final Notes

Double DQN is one of the simplest and most effective fixes in Deep RL.
You keep everything from DQN and change just how the target is computed.

You get better convergence, better performance, and a more robust agent — with minimal effort.

Use Double DQN when:

• Your Q-values explode or diverge
• Rewards are sparse
• You need a stable baseline
• You want to train faster and generalize better

---

References:

• Van Hasselt, H., Guez, A., & Silver, D. (2015). Deep Reinforcement Learning with Double Q-learning. arXiv.
• Van Hasselt, H. (2010). Double Q-learning. Advances in Neural Information Processing Systems.
• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. MIT Press.
• OpenAI Gym documentation.

---

Deep Q Network (DQN) << Previous | Next >> Dueling DQN