lab_reinforcement_learning/questions.md

Questions

Checkpoint 1

1. How do you decide where to move in BabySnake? Explain how to choose moves in enough detail that someone else could follow your instructions.

Checkpoint 2

2. How many distinct states are there for BabySnake on a 4×4 grid? If we assume that all four arrow keys are valid actions in every state, how many rows would the full Q-table contain?

3. The discount factor γ (gamma) can range from 0 to 1. What would be the effect of setting γ to 0? What about 1?

4. The learning rate α (alpha) can also range from 0 to 1. What would be the effect of setting α to 0? What about 1?

5. Calculate the new Q-value for ((2, 2, 3, 3), RIGHT). Explain your answer.

Checkpoint 3

6. At what episode did the agent start reliably finding food?

7. Add print(Q) to train_babysnake.py before the watch call and run it again. Can you read the policy? For a given state, does the highest Q-value point toward the food?

8. How does the trained agent's behavior compare to the reasoning you wrote down in question 1?

Checkpoint 4

9. Full board (runs/snake-v1, ep_5000): Describe the agent's behavior. Does it seem to know where the apple is? Does it move randomly or with some purpose?

10. Features only (runs/snake-v2, ep_3000): How does this agent differ from the v1 agent? What is it doing better? What is it doing that leads to shorter episodes?

11. Final run, early (runs/snake, ep_1300): This agent uses the egocentric view plus apple_dx/apple_dy. What has it learned that neither v1 nor v2 showed?

12. Final run, mature (runs/snake, ep_20000): What does this agent do well? Where does it still make mistakes?

13. In the features-only run (v2), reward rose as episodes got shorter. Why does a snake agent that is getting better at finding apples end up with shorter episodes?

14. The egocentric view crops the observation to a 17×17 window centered on the snake's head. What did the agent gain from this change, and what information did it lose access to?

Checkpoint 5

Answer these questions after completing both training experiments in "Training Frogger."

15. Hypothesis (Attempt 1): Before training, predict what will happen. Will the agent learn to reach the top of the board? What challenge do you think it will face?

16. Evidence (Attempt 1): Copy the first three and last three lines of runs/frogger/training.log. Did training go as expected?

17. Analysis (Attempt 1): What did the agent learn to do? Where did it struggle?

18. Experiment (Attempt 2): What one thing did you change? Write your prediction, show the evidence (first and last few log lines), and describe what happened.

19. Which attempt produced the best agent? What would you try next if you had more time?