Exploring or Exploiting? Understanding Epsilon-Greedy

This page was last edited on 12 November 2025

Epsilon-greedy is a selection strategy that balances exploration (trying new actions) and exploitation (choosing the best-known action). It choose between:

• the best known action (exploitation) with a high probability. 
• a random action (exploration) with a low probability.

We use Epsilon-greedy to avoid getting stuck in local optima. Without exploration, the agent most probably learns about a few actions and never tries better ones.

With Epsilon-greedy, we ensure the agent keeps exploring the environment, especially in the early stages of training. It’s a simple but effective way to improve learning and generalization.

Epsilon-greedy equations

There are two key equations:

1. Action Selection:

   

2. Epsilon Decay (optional, during training):

   

Where:

• ε: exploration rate.
• ε_start​: initial exploration rate (e.g., 1.0).
• ε_min​: minimum value (e.g., 0.01).
• decay: factor (e.g., 0.99).
• t: time step (or episode number).
• Q(s_t​, a): current estimate of the value of taking action a in state s_t

ANALOGY

I’m in a new city and want to find the best restaurant.

• On the first day, I tried random places (exploration).
• After a week, I found a great one and kept going there (exploitation).

But now and then, I still want to try a new one in case it’s better (epsilon-greedy).
So epsilon is like how adventurous I want to be:

• High epsilon = I want to try new places often.
• Low epsilon = I stick with my favorite restaurants.

Inputs and outputs of epsilon-greedy

Inputs:

• Current Q-values: Q(s, a).
• Epsilon ε.
• Action space A.

Output:

• Selected action a_t

EXAMPLE: How epsilon-greedy works

Assumptions:

• Action space A={a₁, a₂, a₃}
• Q-values:
  • Q(s,a₁​)= 5
  • Q(s,a₂​)= 2
  • Q(s,a₃​)= 0
• ε= 0.4 (fixed, no decay)
• Random seed fixed to simulate randomness.

We’ll generate random numbers to simulate exploration:

• If random < ε → explore
• Else → exploit (choose action with highest Q-value)

ITERATION 1:

• Random = 0.25 → explore
• Choose randomly → let’s say a₃

---

ITERATION 2:

• Random = 0.85 → exploit
• Choose arg⁡ max ⁡Q = a₁

---

ITERATION 3:

• Random = 0.33 → explore
• Choose randomly → let’s say a₂

---

ITERATION 4:

• Random = 0.92 → exploit
• Choose a₁

---

ITERATION 5:

• Random = 0.10 → explore
• Choose randomly → let’s say a₂

---

Iteration	Random Value	Decision	Action Chosen
1	0.25	Explore	a3
2	0.85	Exploit	a1
3	0.33	Explore	a2
4	0.92	Exploit	a1
5	0.10	Explore	a2

Interpretation:

• 3 out of 5 actions were exploratory (a₂, a₃)
• 2 out of 5 were greedy (a₁)
• This aligns with ε = 0.4 (40% exploration expected)

---

References:

• Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. MIT Press.
• David Silver (2015). Lectures on Reinforcement Learning.
• “Welcome to Spinning Up in Deep RL!” – Open AI.

---

Choosing RL Algorithm << Previous | Next >> SIM2REAL