edbf76071e
The egocentric/egocentric_player/egocentric_radius flags were removed from retro_gamer in favor of an explicit observation_function. snake_observation.py reproduces the old egocentric+board+extras behavior by calling egocentric_board/encode_board/encode_state directly, and runs/snake/config.toml now points at it. That config change made the prior 12,000-episode checkpoint history incompatible (retro_gamer's checkpoint compatibility checker can't verify a new observation_function is behaviorally equivalent to the old flags, so it conservatively refuses to resume), so the old checkpoints were deleted and a fresh 20,000-episode run was recorded. Track only the four checkpoints the lab actually references (ep_1300, ep_2300, ep_4000, ep_20000) instead of all 200, and update .gitignore so future student runs of runs/snake aren't committed by default. snake_training.md's Q5 training-curve table and Q6 checkpoint episodes are updated to match the real numbers from this run.
4.2 KiB
Snake Training: Conceptual Questions
Answer each question in the space provided. Use evidence from the training log and your observations of the agent at different checkpoints to support your answers.
1. Feature selection
In the first training attempt, the agent received the full 32×16 game board as its input (6 × 32 × 16 = 3,072 numbers). The agent could see every character on the board, yet it never learned to reliably find the apple after 45,000 episodes.
When we added apple_dx and apple_dy — two numbers that encode the direction
from the snake's head to the apple — performance improved dramatically within
hundreds of episodes.
Why didn't the board encoding help the agent find the apple? What did the two new features provide that the board encoding could not?
Your answer:
2. Dimensionality reduction
In the full-board experiment, the agent processed 3,072 input values. When we switched to the egocentric view (a 17×17 window centered on the snake's head), the board input shrank to 17 × 17 × 6 = 1,734 values.
How many input values did the egocentric view save compared to the full board? What is one thing the agent gained from this change, and one thing it lost?
Your answer:
3. Exploration vs. exploitation
With epsilon_decay = 0.995, epsilon falls from 1.0 to 0.05 by episode ~450.
With epsilon_decay = 0.9997 (used in the final run), epsilon is still 0.55 at
episode 2,000.
Sketch a rough curve of epsilon over time for each setting. Why does slower decay produce a better-trained agent, even though it means the agent takes more random actions overall?
Your answer:
4. Runaway loss
In one intermediate experiment, the loss grew from around 35 to hundreds of thousands within a few hundred episodes:
[ep_0300] avg_loss=48.7 avg_reward=+8.1
[ep_0500] avg_loss=347 avg_reward=+12.4
[ep_0700] avg_loss=4,102 avg_reward=+6.5
[ep_1100] avg_loss=686,000 avg_reward=-3.1
This happened because the learning algorithm was using MSE (mean squared error) loss, which is quadratic — an error of size 2 produces a loss of 4, an error of size 10 produces a loss of 100.
Describe the feedback loop that caused the loss to spiral upward. Why does Huber loss (which is linear for large errors) break this cycle?
Your answer:
5. Interpreting the training curve
Look at runs/snake/training.log. The reward climbs, then dips, then climbs
again:
[ep_1300] avg_reward= +31.6 avg_steps=49
[ep_2300] avg_reward= -3.8 avg_steps=87
[ep_4000] avg_reward= +53.2 avg_steps=42
[ep_20000] avg_reward=+335.6 avg_steps=94
Notice that around episode 4,000, avg_steps dropped sharply (from ~87 to 42) at the same time reward jumped. Then by episode 20,000, steps rose again while reward kept climbing.
What do you think the agent was doing at each of these stages? Use the avg_steps and avg_reward numbers to support your interpretation.
Your answer:
6. Policy observation
Run these commands to watch the agent at three checkpoints:
retro-gamer play runs/snake --checkpoint ep_1300
retro-gamer play runs/snake --checkpoint ep_4000
retro-gamer play runs/snake --checkpoint ep_20000
Describe the agent's behavior at each checkpoint. What has the agent learned by episode 4,000 that it hadn't yet learned at episode 1,300? What does the episode 20,000 agent do that the earlier agents do not?
ep_1300:
ep_4000:
ep_20000:
7. CNN vs. MLP
In the first attempt (full board, no explicit features), we used a CNN
(spatial = true). In the final run (egocentric board + explicit features), we
used an MLP (spatial = false).
Why might an MLP be a reasonable choice when using the egocentric view, even though the input is still a 2D board? What does the CNN offer that the MLP does not, and why is that less important with an egocentric observation?
Your answer:
8. Hyperparameter comparison
Suppose you ran two otherwise identical training experiments:
- Run A:
learning_rate = 0.001 - Run B:
learning_rate = 0.0001
Based on what you learned from the runaway loss in Question 4, predict what would happen in each run. What does this tell you about the trade-off when choosing a learning rate?
Your answer: