mwc/lab_reinforcement_learning
Template
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Migrate snake example to observation_function and refresh its training run

Browse Source 
The egocentric/egocentric_player/egocentric_radius flags were removed
from retro_gamer in favor of an explicit observation_function.
snake_observation.py reproduces the old egocentric+board+extras
behavior by calling egocentric_board/encode_board/encode_state
directly, and runs/snake/config.toml now points at it.

That config change made the prior 12,000-episode checkpoint history
incompatible (retro_gamer's checkpoint compatibility checker can't
verify a new observation_function is behaviorally equivalent to the
old flags, so it conservatively refuses to resume), so the old
checkpoints were deleted and a fresh 20,000-episode run was recorded.
Track only the four checkpoints the lab actually references
(ep_1300, ep_2300, ep_4000, ep_20000) instead of all 200, and update
.gitignore so future student runs of runs/snake aren't committed by
default. snake_training.md's Q5 training-curve table and Q6 checkpoint
episodes are updated to match the real numbers from this run.
This commit is contained in:
Chris Proctor
2026-06-24 07:50:41 -04:00
parent e8a24ae7be
commit edbf76071e
9 changed files with 322 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
snake_training.md
View File

@@ -74,17 +74,18 @@ Huber loss (which is linear for large errors) break this cycle?**
## 5. Interpreting the training curve
Look at the snake training log. The reward climbs, then dips, then climbs again:
Look at `runs/snake/training.log`. The reward climbs, then dips, then climbs
again:
```
[ep_1100] avg_reward=+34.5 avg_steps=57
[ep_1800] avg_reward=+4.4 avg_steps=98
[ep_3800] avg_reward=+51.2 avg_steps=33
[ep_9000] avg_reward=+246.0 avg_steps=85
[ep_1300] avg_reward= +31.6 avg_steps=49
[ep_2300] avg_reward= -3.8 avg_steps=87
[ep_4000] avg_reward= +53.2 avg_steps=42
[ep_20000] avg_reward=+335.6 avg_steps=94
```
Notice that around episode 3,800, avg_steps dropped sharply (from ~98 to 33)
at the same time reward jumped. Then by episode 9,000, steps rose again while
Notice that around episode 4,000, avg_steps dropped sharply (from ~87 to 42)
at the same time reward jumped. Then by episode 20,000, steps rose again while
reward kept climbing.
**What do you think the agent was doing at each of these stages? Use the
@@ -99,20 +100,20 @@ avg_steps and avg_reward numbers to support your interpretation.**
Run these commands to watch the agent at three checkpoints:
```
retro-gamer play runs/snake --checkpoint ep_1100
retro-gamer play runs/snake --checkpoint ep_5400
retro-gamer play runs/snake --checkpoint ep_17100
retro-gamer play runs/snake --checkpoint ep_1300
retro-gamer play runs/snake --checkpoint ep_4000
retro-gamer play runs/snake --checkpoint ep_20000
```
**Describe the agent's behavior at each checkpoint. What has the agent learned
by episode 5,400 that it hadn't yet learned at episode 1,100? What does the
episode 17,100 agent do that the earlier agents do not?**
by episode 4,000 that it hadn't yet learned at episode 1,300? What does the
episode 20,000 agent do that the earlier agents do not?**
*ep_1100:*
*ep_1300:*
*ep_5400:*
*ep_4000:*
*ep_17100:*
*ep_20000:*
---