edbf76071e
The egocentric/egocentric_player/egocentric_radius flags were removed from retro_gamer in favor of an explicit observation_function. snake_observation.py reproduces the old egocentric+board+extras behavior by calling egocentric_board/encode_board/encode_state directly, and runs/snake/config.toml now points at it. That config change made the prior 12,000-episode checkpoint history incompatible (retro_gamer's checkpoint compatibility checker can't verify a new observation_function is behaviorally equivalent to the old flags, so it conservatively refuses to resume), so the old checkpoints were deleted and a fresh 20,000-episode run was recorded. Track only the four checkpoints the lab actually references (ep_1300, ep_2300, ep_4000, ep_20000) instead of all 200, and update .gitignore so future student runs of runs/snake aren't committed by default. snake_training.md's Q5 training-curve table and Q6 checkpoint episodes are updated to match the real numbers from this run.
145 lines
4.2 KiB
Markdown
145 lines
4.2 KiB
Markdown
# Snake Training: Conceptual Questions
|
||
|
||
Answer each question in the space provided. Use evidence from the training log
|
||
and your observations of the agent at different checkpoints to support your answers.
|
||
|
||
---
|
||
|
||
## 1. Feature selection
|
||
|
||
In the first training attempt, the agent received the full 32×16 game board as
|
||
its input (6 × 32 × 16 = 3,072 numbers). The agent could see every character on
|
||
the board, yet it never learned to reliably find the apple after 45,000 episodes.
|
||
|
||
When we added `apple_dx` and `apple_dy` — two numbers that encode the direction
|
||
from the snake's head to the apple — performance improved dramatically within
|
||
hundreds of episodes.
|
||
|
||
**Why didn't the board encoding help the agent find the apple? What did the two
|
||
new features provide that the board encoding could not?**
|
||
|
||
*Your answer:*
|
||
|
||
---
|
||
|
||
## 2. Dimensionality reduction
|
||
|
||
In the full-board experiment, the agent processed 3,072 input values. When we
|
||
switched to the egocentric view (a 17×17 window centered on the snake's head),
|
||
the board input shrank to 17 × 17 × 6 = 1,734 values.
|
||
|
||
**How many input values did the egocentric view save compared to the full board?
|
||
What is one thing the agent gained from this change, and one thing it lost?**
|
||
|
||
*Your answer:*
|
||
|
||
---
|
||
|
||
## 3. Exploration vs. exploitation
|
||
|
||
With `epsilon_decay = 0.995`, epsilon falls from 1.0 to 0.05 by episode ~450.
|
||
With `epsilon_decay = 0.9997` (used in the final run), epsilon is still 0.55 at
|
||
episode 2,000.
|
||
|
||
**Sketch a rough curve of epsilon over time for each setting. Why does slower
|
||
decay produce a better-trained agent, even though it means the agent takes more
|
||
random actions overall?**
|
||
|
||
*Your answer:*
|
||
|
||
---
|
||
|
||
## 4. Runaway loss
|
||
|
||
In one intermediate experiment, the loss grew from around 35 to hundreds of
|
||
thousands within a few hundred episodes:
|
||
|
||
```
|
||
[ep_0300] avg_loss=48.7 avg_reward=+8.1
|
||
[ep_0500] avg_loss=347 avg_reward=+12.4
|
||
[ep_0700] avg_loss=4,102 avg_reward=+6.5
|
||
[ep_1100] avg_loss=686,000 avg_reward=-3.1
|
||
```
|
||
|
||
This happened because the learning algorithm was using MSE (mean squared error)
|
||
loss, which is *quadratic* — an error of size 2 produces a loss of 4, an error
|
||
of size 10 produces a loss of 100.
|
||
|
||
**Describe the feedback loop that caused the loss to spiral upward. Why does
|
||
Huber loss (which is linear for large errors) break this cycle?**
|
||
|
||
*Your answer:*
|
||
|
||
---
|
||
|
||
## 5. Interpreting the training curve
|
||
|
||
Look at `runs/snake/training.log`. The reward climbs, then dips, then climbs
|
||
again:
|
||
|
||
```
|
||
[ep_1300] avg_reward= +31.6 avg_steps=49
|
||
[ep_2300] avg_reward= -3.8 avg_steps=87
|
||
[ep_4000] avg_reward= +53.2 avg_steps=42
|
||
[ep_20000] avg_reward=+335.6 avg_steps=94
|
||
```
|
||
|
||
Notice that around episode 4,000, avg_steps dropped sharply (from ~87 to 42)
|
||
at the same time reward jumped. Then by episode 20,000, steps rose again while
|
||
reward kept climbing.
|
||
|
||
**What do you think the agent was doing at each of these stages? Use the
|
||
avg_steps and avg_reward numbers to support your interpretation.**
|
||
|
||
*Your answer:*
|
||
|
||
---
|
||
|
||
## 6. Policy observation
|
||
|
||
Run these commands to watch the agent at three checkpoints:
|
||
|
||
```
|
||
retro-gamer play runs/snake --checkpoint ep_1300
|
||
retro-gamer play runs/snake --checkpoint ep_4000
|
||
retro-gamer play runs/snake --checkpoint ep_20000
|
||
```
|
||
|
||
**Describe the agent's behavior at each checkpoint. What has the agent learned
|
||
by episode 4,000 that it hadn't yet learned at episode 1,300? What does the
|
||
episode 20,000 agent do that the earlier agents do not?**
|
||
|
||
*ep_1300:*
|
||
|
||
*ep_4000:*
|
||
|
||
*ep_20000:*
|
||
|
||
---
|
||
|
||
## 7. CNN vs. MLP
|
||
|
||
In the first attempt (full board, no explicit features), we used a CNN
|
||
(`spatial = true`). In the final run (egocentric board + explicit features), we
|
||
used an MLP (`spatial = false`).
|
||
|
||
**Why might an MLP be a reasonable choice when using the egocentric view, even
|
||
though the input is still a 2D board? What does the CNN offer that the MLP does
|
||
not, and why is that less important with an egocentric observation?**
|
||
|
||
*Your answer:*
|
||
|
||
---
|
||
|
||
## 8. Hyperparameter comparison
|
||
|
||
Suppose you ran two otherwise identical training experiments:
|
||
- Run A: `learning_rate = 0.001`
|
||
- Run B: `learning_rate = 0.0001`
|
||
|
||
**Based on what you learned from the runaway loss in Question 4, predict what
|
||
would happen in each run. What does this tell you about the trade-off when
|
||
choosing a learning rate?**
|
||
|
||
*Your answer:*
|