Dueling DQN

This page was last edited on 15 May 2025

Dueling DQN is a variant of Deep Q-Networks.
Instead of learning just Q-values, it splits the Q-function into two parts:

• Value function V(s) – how good is the state
• Advantage function A(s, a) – how good is a specific action in that state

Then it combines them into Q(s, a).

Why Choose Dueling DQN?

In many states, the choice of action doesn’t matter much.
Dueling DQN helps by separately learning:

• How valuable the state is
• Which actions are better in that state

This improves learning efficiency and policy quality, especially in states with many similar-valued actions.

What Type of Learning Is It?

Value-based deep reinforcement learning. It estimates the Q-value for every action, using a shared network.

Model-Free or Model-Based?

Model-Free.
It learns directly from experience, no model of the environment.

What Is It Trying to Compute?

It tries to compute the optimal Q-function, same as DQN:

   

Where:

• Q(s, a) – estimated value of taking action a in state s.
• V(s) – value of being in state s, regardless of which action is taken.
• A(s, a) – advantage of taking action a in state s; it tells how much better that action is compared to the average.
• meanₐ A(s, a) – the average advantage over all possible actions in state s, used to normalize the advantages.

The subtraction of mean(A) keeps the Q-values centered and stable.

Training Loop

Same high-level flow as DQN / Double DQN:

1. Collect experience (s, a, r, s', done)
2. Store in replay buffer
3. Sample mini-batch
4. Compute Double DQN-style target
5. Compute Q(s, a) using dueling architecture
6. Compute loss between predicted and target Q
7. Backpropagate and update
8. Update target network (soft or hard copy)

How to Implement Dueling DQN?

Instead of outputting Q-values directly, the network has two output heads:

• One for V(s) – a single scalar
• One for A(s, a) – a vector for all actions

Then combine:

Q = V + (A - A.mean(dim=1, keepdim=True))

You can use this in both DQN and Double DQN.

On-Policy or Off-Policy?

Off-Policy.
Learns from replayed past experiences, not just the current policy.

Exploration vs. Exploitation?

Uses standard ε-greedy exploration:

• With probability ε → random action
• Otherwise → action with max Q

ε is decayed over time.

When Does It Converge?

• With stable updates (small learning rate, replay buffer, target network)
• With enough exploration
• Works better when action effects are subtle but state value matters

Where Does It Struggle?

• In very small action spaces, the benefit is minimal
• Poor implementation of the architecture (e.g., wrong normalization)
• If A dominates V, training becomes unstable

What Problems Is It Good For?

• Large action spaces
• Visual or noisy inputs (Atari, image-based input)
• When state value is more important than the exact action choice

Common Traps and Mistakes

• Forgetting to normalize the advantage term → unstable Q-values
• Not using it with Double DQN → more overestimation
• Confusing architecture with Actor-Critic (this is still Q-learning)

Dueling DQN Equation

   

Where:

• Q(s, a) – estimated Q-value
• V(s) – value of being in state s
• A(s, a) – advantage of action a in state s
• mean_a A(s, a) – average advantage over all actions (to avoid redundancy)

This ensures the Q-values are identifiable – meaning the network can learn something useful, not just shift the values arbitrarily.

---

References:

• Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2016). Dueling Network Architectures for Deep Reinforcement Learning. Proceedings of the 33rd International Conference on Machine Learning (ICML), 48, 1995–2003.
• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. MIT Press.
• OpenAI Gym documentation.

---

Double DQN << Previous | Next >> Proximal Policy Optimization (PPO)