4.2 KiB
Snake Training: Conceptual Questions
Answer each question in the space provided. Use evidence from the training log and your observations of the agent at different checkpoints to support your answers.
1. Feature selection
In the first training attempt, the agent received the full 32×16 game board as its input (6 × 32 × 16 = 3,072 numbers). The agent could see every character on the board, yet it never learned to reliably find the apple after 45,000 episodes.
When we added apple_dx and apple_dy — two numbers that encode the direction
from the snake's head to the apple — performance improved dramatically within
hundreds of episodes.
Why didn't the board encoding help the agent find the apple? What did the two new features provide that the board encoding could not?
Your answer:
2. Dimensionality reduction
In the full-board experiment, the agent processed 3,072 input values. When we switched to the egocentric view (a 17×17 window centered on the snake's head), the board input shrank to 17 × 17 × 6 = 1,734 values.
How many input values did the egocentric view save compared to the full board? What is one thing the agent gained from this change, and one thing it lost?
Your answer:
3. Exploration vs. exploitation
With epsilon_decay = 0.995, epsilon falls from 1.0 to 0.05 by episode ~450.
With epsilon_decay = 0.9997 (used in the final run), epsilon is still 0.55 at
episode 2,000.
Sketch a rough curve of epsilon over time for each setting. Why does slower decay produce a better-trained agent, even though it means the agent takes more random actions overall?
Your answer:
4. Runaway loss
In one intermediate experiment, the loss grew from around 35 to hundreds of thousands within a few hundred episodes:
[ep_0300] avg_loss=48.7 avg_reward=+8.1
[ep_0500] avg_loss=347 avg_reward=+12.4
[ep_0700] avg_loss=4,102 avg_reward=+6.5
[ep_1100] avg_loss=686,000 avg_reward=-3.1
This happened because the learning algorithm was using MSE (mean squared error) loss, which is quadratic — an error of size 2 produces a loss of 4, an error of size 10 produces a loss of 100.
Describe the feedback loop that caused the loss to spiral upward. Why does Huber loss (which is linear for large errors) break this cycle?
Your answer:
5. Interpreting the training curve
Look at the snake training log. The reward climbs, then dips, then climbs again:
[ep_1100] avg_reward=+34.5 avg_steps=57
[ep_1800] avg_reward=+4.4 avg_steps=98
[ep_3800] avg_reward=+51.2 avg_steps=33
[ep_9000] avg_reward=+246.0 avg_steps=85
Notice that around episode 3,800, avg_steps dropped sharply (from ~98 to 33) at the same time reward jumped. Then by episode 9,000, steps rose again while reward kept climbing.
What do you think the agent was doing at each of these stages? Use the avg_steps and avg_reward numbers to support your interpretation.
Your answer:
6. Policy observation
Run these commands to watch the agent at three checkpoints:
retro-gamer play runs/snake --checkpoint ep_1100
retro-gamer play runs/snake --checkpoint ep_5400
retro-gamer play runs/snake --checkpoint ep_17100
Describe the agent's behavior at each checkpoint. What has the agent learned by episode 5,400 that it hadn't yet learned at episode 1,100? What does the episode 17,100 agent do that the earlier agents do not?
ep_1100:
ep_5400:
ep_17100:
7. CNN vs. MLP
In the first attempt (full board, no explicit features), we used a CNN
(spatial = true). In the final run (egocentric board + explicit features), we
used an MLP (spatial = false).
Why might an MLP be a reasonable choice when using the egocentric view, even though the input is still a 2D board? What does the CNN offer that the MLP does not, and why is that less important with an egocentric observation?
Your answer:
8. Hyperparameter comparison
Suppose you ran two otherwise identical training experiments:
- Run A:
learning_rate = 0.001 - Run B:
learning_rate = 0.0001
Based on what you learned from the runaway loss in Question 4, predict what would happen in each run. What does this tell you about the trade-off when choosing a learning rate?
Your answer: