mwc/lab_reinforcement_learning
Template
3
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Migrate snake example to observation_function and refresh its training run

Browse Source 
The egocentric/egocentric_player/egocentric_radius flags were removed
from retro_gamer in favor of an explicit observation_function.
snake_observation.py reproduces the old egocentric+board+extras
behavior by calling egocentric_board/encode_board/encode_state
directly, and runs/snake/config.toml now points at it.

That config change made the prior 12,000-episode checkpoint history
incompatible (retro_gamer's checkpoint compatibility checker can't
verify a new observation_function is behaviorally equivalent to the
old flags, so it conservatively refuses to resume), so the old
checkpoints were deleted and a fresh 20,000-episode run was recorded.
Track only the four checkpoints the lab actually references
(ep_1300, ep_2300, ep_4000, ep_20000) instead of all 200, and update
.gitignore so future student runs of runs/snake aren't committed by
default. snake_training.md's Q5 training-curve table and Q6 checkpoint
episodes are updated to match the real numbers from this run.
This commit is contained in:
Chris Proctor
2026-06-24 07:50:41 -04:00
parent e8a24ae7be
commit edbf76071e
9 changed files with 322 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

213
runs/snake/training.log Normal file
View File

@@ -0,0 +1,213 @@
[INIT] === Network Architecture ===
[INIT] Board: 17×17, character set: 6 chars (one-hot per cell)
[INIT] Observed state features: 2 | Actions (incl. no-op): 5
[INIT] spatial=False → using MLP architecture
[INIT] Rationale: the board encodes UI/status rather than a spatial scene;
[INIT] a flat MLP over the full observation is sufficient.
[INIT] MLP: 1736 → 128 → 64 → 5
[INIT] Hidden layers: 2 | Layer sizes: [128, 64]
[INIT] Output: 5 Q-values
[INIT] Actions: ['KEY_RIGHT', 'KEY_UP', 'KEY_LEFT', 'KEY_DOWN'] + (no-op)
[INIT] Device: mps
=== Training started | 2026-06-23 22:39:07 ===
[ep_0100] ep=0001-0100 avg_reward=-7.8 avg_steps=49 epsilon=0.970 avg_loss=0.8 time=0m06s total=0m06s
[ep_0200] ep=0101-0200 avg_reward=-5.7 avg_steps=51 epsilon=0.942 avg_loss=0.9 time=0m07s total=0m13s
[ep_0300] ep=0201-0300 avg_reward=+0.1 avg_steps=49 epsilon=0.914 avg_loss=1.2 time=0m06s total=0m20s
[ep_0400] ep=0301-0400 avg_reward=+7.9 avg_steps=70 epsilon=0.887 avg_loss=1.6 time=0m09s total=0m30s
[ep_0500] ep=0401-0500 avg_reward=+12.8 avg_steps=57 epsilon=0.861 avg_loss=1.9 time=0m08s total=0m38s
[ep_0600] ep=0501-0600 avg_reward=+20.5 avg_steps=57 epsilon=0.835 avg_loss=2.3 time=0m08s total=0m47s
[ep_0700] ep=0601-0700 avg_reward=+23.7 avg_steps=62 epsilon=0.811 avg_loss=2.8 time=0m10s total=0m58s
[ep_0800] ep=0701-0800 avg_reward=+21.2 avg_steps=56 epsilon=0.787 avg_loss=3.4 time=0m10s total=1m08s
[ep_0900] ep=0801-0900 avg_reward=+30.1 avg_steps=55 epsilon=0.763 avg_loss=4.1 time=0m10s total=1m19s
[ep_1000] ep=0901-1000 avg_reward=+33.6 avg_steps=59 epsilon=0.741 avg_loss=4.6 time=0m11s total=1m31s
[ep_1100] ep=1001-1100 avg_reward=+36.0 avg_steps=58 epsilon=0.719 avg_loss=5.4 time=0m11s total=1m43s
[ep_1200] ep=1101-1200 avg_reward=+34.0 avg_steps=50 epsilon=0.698 avg_loss=6.1 time=0m10s total=1m53s
[ep_1300] ep=1201-1300 avg_reward=+31.6 avg_steps=49 epsilon=0.677 avg_loss=6.7 time=0m09s total=2m03s
[ep_1400] ep=1301-1400 avg_reward=+30.4 avg_steps=60 epsilon=0.657 avg_loss=7.2 time=0m12s total=2m15s
[ep_1500] ep=1401-1500 avg_reward=+28.0 avg_steps=65 epsilon=0.638 avg_loss=7.6 time=0m13s total=2m28s
[ep_1600] ep=1501-1600 avg_reward=+16.3 avg_steps=68 epsilon=0.619 avg_loss=7.6 time=0m14s total=2m42s
[ep_1700] ep=1601-1700 avg_reward=+10.1 avg_steps=76 epsilon=0.600 avg_loss=7.7 time=0m18s total=3m01s
[ep_1800] ep=1701-1800 avg_reward=+6.9 avg_steps=89 epsilon=0.583 avg_loss=7.6 time=0m20s total=3m21s
[ep_1900] ep=1801-1900 avg_reward=+1.9 avg_steps=85 epsilon=0.565 avg_loss=7.4 time=0m16s total=3m38s
[ep_2000] ep=1901-2000 avg_reward=+1.1 avg_steps=93 epsilon=0.549 avg_loss=6.9 time=0m18s total=3m57s
[ep_2100] ep=2001-2100 avg_reward=+1.4 avg_steps=105 epsilon=0.533 avg_loss=6.2 time=0m21s total=4m18s
[ep_2200] ep=2101-2200 avg_reward=-0.8 avg_steps=83 epsilon=0.517 avg_loss=5.5 time=0m16s total=4m34s
[ep_2300] ep=2201-2300 avg_reward=-3.8 avg_steps=87 epsilon=0.502 avg_loss=5.1 time=0m17s total=4m52s
[ep_2400] ep=2301-2400 avg_reward=-0.1 avg_steps=89 epsilon=0.487 avg_loss=4.6 time=0m17s total=5m09s
[ep_2500] ep=2401-2500 avg_reward=-2.5 avg_steps=94 epsilon=0.472 avg_loss=4.1 time=0m18s total=5m28s
[ep_2600] ep=2501-2600 avg_reward=+1.2 avg_steps=97 epsilon=0.458 avg_loss=3.7 time=0m19s total=5m48s
[ep_2700] ep=2601-2700 avg_reward=+8.0 avg_steps=85 epsilon=0.445 avg_loss=3.4 time=0m16s total=6m05s
[ep_2800] ep=2701-2800 avg_reward=+5.2 avg_steps=83 epsilon=0.432 avg_loss=2.9 time=0m16s total=6m21s
[ep_2900] ep=2801-2900 avg_reward=+8.4 avg_steps=89 epsilon=0.419 avg_loss=2.7 time=0m17s total=6m39s
[ep_3000] ep=2901-3000 avg_reward=+4.0 avg_steps=98 epsilon=0.407 avg_loss=2.4 time=0m19s total=6m59s
[ep_3100] ep=3001-3100 avg_reward=+6.0 avg_steps=80 epsilon=0.394 avg_loss=2.1 time=0m16s total=7m15s
[ep_3200] ep=3101-3200 avg_reward=+7.0 avg_steps=86 epsilon=0.383 avg_loss=2.0 time=0m17s total=7m33s
[ep_3300] ep=3201-3300 avg_reward=+4.3 avg_steps=93 epsilon=0.372 avg_loss=1.8 time=0m19s total=7m52s
[ep_3400] ep=3301-3400 avg_reward=+2.7 avg_steps=98 epsilon=0.361 avg_loss=1.8 time=0m20s total=8m12s
[ep_3500] ep=3401-3500 avg_reward=+7.7 avg_steps=99 epsilon=0.350 avg_loss=1.6 time=0m20s total=8m32s
[ep_3600] ep=3501-3600 avg_reward=+21.0 avg_steps=86 epsilon=0.340 avg_loss=1.5 time=0m17s total=8m50s
[ep_3700] ep=3601-3700 avg_reward=+30.8 avg_steps=67 epsilon=0.330 avg_loss=1.4 time=0m13s total=9m03s
[ep_3800] ep=3701-3800 avg_reward=+38.4 avg_steps=53 epsilon=0.320 avg_loss=1.3 time=0m10s total=9m14s
[ep_3900] ep=3801-3900 avg_reward=+48.0 avg_steps=49 epsilon=0.310 avg_loss=1.4 time=0m09s total=9m24s
[ep_4000] ep=3901-4000 avg_reward=+53.2 avg_steps=42 epsilon=0.301 avg_loss=1.4 time=0m08s total=9m33s
[ep_4100] ep=4001-4100 avg_reward=+49.1 avg_steps=47 epsilon=0.292 avg_loss=1.5 time=0m09s total=9m42s
[ep_4200] ep=4101-4200 avg_reward=+49.9 avg_steps=50 epsilon=0.284 avg_loss=1.6 time=0m10s total=9m53s
[ep_4300] ep=4201-4300 avg_reward=+54.5 avg_steps=48 epsilon=0.275 avg_loss=1.7 time=0m09s total=10m02s
[ep_4400] ep=4301-4400 avg_reward=+57.5 avg_steps=48 epsilon=0.267 avg_loss=1.8 time=0m09s total=10m12s
[ep_4500] ep=4401-4500 avg_reward=+53.6 avg_steps=47 epsilon=0.259 avg_loss=1.9 time=0m09s total=10m22s
[ep_4600] ep=4501-4600 avg_reward=+59.3 avg_steps=44 epsilon=0.252 avg_loss=1.9 time=0m09s total=10m31s
[ep_4700] ep=4601-4700 avg_reward=+60.2 avg_steps=39 epsilon=0.244 avg_loss=1.8 time=0m08s total=10m39s
[ep_4800] ep=4701-4800 avg_reward=+58.6 avg_steps=40 epsilon=0.237 avg_loss=1.8 time=0m08s total=10m47s
[ep_4900] ep=4801-4900 avg_reward=+64.5 avg_steps=54 epsilon=0.230 avg_loss=1.9 time=0m11s total=10m59s
[ep_5000] ep=4901-5000 avg_reward=+72.9 avg_steps=37 epsilon=0.223 avg_loss=1.9 time=0m07s total=11m06s
[ep_5100] ep=5001-5100 avg_reward=+72.2 avg_steps=36 epsilon=0.216 avg_loss=1.9 time=0m07s total=11m14s
[ep_5200] ep=5101-5200 avg_reward=+71.6 avg_steps=39 epsilon=0.210 avg_loss=1.9 time=0m08s total=11m22s
[ep_5300] ep=5201-5300 avg_reward=+67.9 avg_steps=34 epsilon=0.204 avg_loss=1.9 time=0m07s total=11m29s
[ep_5400] ep=5301-5400 avg_reward=+89.2 avg_steps=43 epsilon=0.198 avg_loss=1.9 time=0m09s total=11m38s
[ep_5500] ep=5401-5500 avg_reward=+92.0 avg_steps=42 epsilon=0.192 avg_loss=2.0 time=0m08s total=11m47s
[ep_5600] ep=5501-5600 avg_reward=+86.4 avg_steps=35 epsilon=0.186 avg_loss=2.1 time=0m07s total=11m55s
[ep_5700] ep=5601-5700 avg_reward=+92.5 avg_steps=37 epsilon=0.181 avg_loss=2.2 time=0m07s total=12m02s
[ep_5800] ep=5701-5800 avg_reward=+98.5 avg_steps=45 epsilon=0.175 avg_loss=2.3 time=0m09s total=12m12s
[ep_5900] ep=5801-5900 avg_reward=+100.9 avg_steps=39 epsilon=0.170 avg_loss=2.5 time=0m08s total=12m20s
[ep_6000] ep=5901-6000 avg_reward=+95.3 avg_steps=40 epsilon=0.165 avg_loss=2.7 time=0m08s total=12m28s
[ep_6100] ep=6001-6100 avg_reward=+92.4 avg_steps=40 epsilon=0.160 avg_loss=2.9 time=0m08s total=12m37s
[ep_6200] ep=6101-6200 avg_reward=+100.2 avg_steps=45 epsilon=0.156 avg_loss=3.1 time=0m09s total=12m46s
[ep_6300] ep=6201-6300 avg_reward=+96.4 avg_steps=43 epsilon=0.151 avg_loss=3.5 time=0m09s total=12m56s
[ep_6400] ep=6301-6400 avg_reward=+107.5 avg_steps=45 epsilon=0.147 avg_loss=3.9 time=0m09s total=13m05s
[ep_6500] ep=6401-6500 avg_reward=+87.4 avg_steps=38 epsilon=0.142 avg_loss=4.1 time=0m08s total=13m13s
[ep_6600] ep=6501-6600 avg_reward=+126.1 avg_steps=51 epsilon=0.138 avg_loss=4.7 time=0m10s total=13m24s
[ep_6700] ep=6601-6700 avg_reward=+118.6 avg_steps=42 epsilon=0.134 avg_loss=5.1 time=0m08s total=13m33s
[ep_6800] ep=6701-6800 avg_reward=+117.7 avg_steps=44 epsilon=0.130 avg_loss=5.7 time=0m09s total=13m42s
[ep_6900] ep=6801-6900 avg_reward=+154.6 avg_steps=56 epsilon=0.126 avg_loss=6.2 time=0m11s total=13m54s
[ep_7000] ep=6901-7000 avg_reward=+129.3 avg_steps=44 epsilon=0.122 avg_loss=6.8 time=0m09s total=14m04s
[ep_7100] ep=7001-7100 avg_reward=+114.2 avg_steps=40 epsilon=0.119 avg_loss=7.2 time=0m08s total=14m12s
[ep_7200] ep=7101-7200 avg_reward=+147.2 avg_steps=52 epsilon=0.115 avg_loss=7.6 time=0m11s total=14m24s
[ep_7300] ep=7201-7300 avg_reward=+122.9 avg_steps=45 epsilon=0.112 avg_loss=8.1 time=0m09s total=14m33s
[ep_7400] ep=7301-7400 avg_reward=+136.2 avg_steps=47 epsilon=0.109 avg_loss=8.4 time=0m10s total=14m43s
[ep_7500] ep=7401-7500 avg_reward=+151.2 avg_steps=51 epsilon=0.105 avg_loss=8.5 time=0m11s total=14m54s
[ep_7600] ep=7501-7600 avg_reward=+150.5 avg_steps=56 epsilon=0.102 avg_loss=8.5 time=0m11s total=15m06s
[ep_7700] ep=7601-7700 avg_reward=+120.3 avg_steps=46 epsilon=0.099 avg_loss=8.6 time=0m09s total=15m16s
[ep_7800] ep=7701-7800 avg_reward=+121.5 avg_steps=47 epsilon=0.096 avg_loss=8.5 time=0m09s total=15m26s
[ep_7900] ep=7801-7900 avg_reward=+160.6 avg_steps=60 epsilon=0.093 avg_loss=8.7 time=0m12s total=15m39s
[ep_8000] ep=7901-8000 avg_reward=+134.2 avg_steps=53 epsilon=0.091 avg_loss=8.6 time=0m11s total=15m50s
[ep_8100] ep=8001-8100 avg_reward=+141.0 avg_steps=51 epsilon=0.088 avg_loss=8.5 time=0m10s total=16m01s
[ep_8200] ep=8101-8200 avg_reward=+167.7 avg_steps=57 epsilon=0.085 avg_loss=8.2 time=0m12s total=16m13s
[ep_8300] ep=8201-8300 avg_reward=+165.9 avg_steps=56 epsilon=0.083 avg_loss=8.0 time=0m11s total=16m25s
[ep_8400] ep=8301-8400 avg_reward=+199.9 avg_steps=71 epsilon=0.080 avg_loss=7.9 time=0m15s total=16m40s
[ep_8500] ep=8401-8500 avg_reward=+196.7 avg_steps=66 epsilon=0.078 avg_loss=7.5 time=0m14s total=16m54s
[ep_8600] ep=8501-8600 avg_reward=+218.4 avg_steps=71 epsilon=0.076 avg_loss=7.4 time=0m15s total=17m10s
[ep_8700] ep=8601-8700 avg_reward=+204.1 avg_steps=74 epsilon=0.074 avg_loss=7.1 time=0m15s total=17m25s
[ep_8800] ep=8701-8800 avg_reward=+216.2 avg_steps=70 epsilon=0.071 avg_loss=6.8 time=0m14s total=17m40s
[ep_8900] ep=8801-8900 avg_reward=+167.1 avg_steps=58 epsilon=0.069 avg_loss=6.7 time=0m12s total=17m53s
[ep_9000] ep=8901-9000 avg_reward=+188.4 avg_steps=58 epsilon=0.067 avg_loss=6.6 time=0m12s total=18m05s
[ep_9100] ep=9001-9100 avg_reward=+225.5 avg_steps=70 epsilon=0.065 avg_loss=6.6 time=0m14s total=18m20s
[ep_9200] ep=9101-9200 avg_reward=+260.5 avg_steps=81 epsilon=0.063 avg_loss=6.5 time=0m17s total=18m37s
[ep_9300] ep=9201-9300 avg_reward=+250.0 avg_steps=77 epsilon=0.061 avg_loss=6.3 time=0m16s total=18m54s
[ep_9400] ep=9301-9400 avg_reward=+313.0 avg_steps=93 epsilon=0.060 avg_loss=6.1 time=0m19s total=19m14s
[ep_9500] ep=9401-9500 avg_reward=+314.2 avg_steps=93 epsilon=0.058 avg_loss=5.9 time=0m19s total=19m34s
[ep_9600] ep=9501-9600 avg_reward=+246.5 avg_steps=75 epsilon=0.056 avg_loss=5.7 time=0m16s total=19m50s
[ep_9700] ep=9601-9700 avg_reward=+274.8 avg_steps=82 epsilon=0.054 avg_loss=5.5 time=0m17s total=20m07s
[ep_9800] ep=9701-9800 avg_reward=+329.7 avg_steps=97 epsilon=0.053 avg_loss=5.5 time=0m20s total=20m28s
[ep_9900] ep=9801-9900 avg_reward=+286.5 avg_steps=87 epsilon=0.051 avg_loss=5.4 time=0m18s total=20m47s
[ep_10000] ep=9901-10000 avg_reward=+304.0 avg_steps=88 epsilon=0.050 avg_loss=5.5 time=0m18s total=21m05s
[ep_10100] ep=10001-10100 avg_reward=+349.1 avg_steps=99 epsilon=0.050 avg_loss=5.5 time=0m21s total=21m26s
[ep_10200] ep=10101-10200 avg_reward=+327.1 avg_steps=93 epsilon=0.050 avg_loss=5.3 time=0m19s total=21m46s
[ep_10300] ep=10201-10300 avg_reward=+347.8 avg_steps=98 epsilon=0.050 avg_loss=5.3 time=0m20s total=22m07s
[ep_10400] ep=10301-10400 avg_reward=+327.0 avg_steps=96 epsilon=0.050 avg_loss=5.1 time=0m20s total=22m28s
[ep_10500] ep=10401-10500 avg_reward=+330.2 avg_steps=96 epsilon=0.050 avg_loss=5.1 time=0m20s total=22m49s
[ep_10600] ep=10501-10600 avg_reward=+335.0 avg_steps=92 epsilon=0.050 avg_loss=5.0 time=0m19s total=23m09s
[ep_10700] ep=10601-10700 avg_reward=+313.9 avg_steps=88 epsilon=0.050 avg_loss=5.0 time=0m19s total=23m28s
[ep_10800] ep=10701-10800 avg_reward=+368.5 avg_steps=104 epsilon=0.050 avg_loss=5.0 time=0m22s total=23m50s
[ep_10900] ep=10801-10900 avg_reward=+366.0 avg_steps=104 epsilon=0.050 avg_loss=4.9 time=0m22s total=24m12s
[ep_11000] ep=10901-11000 avg_reward=+309.2 avg_steps=86 epsilon=0.050 avg_loss=4.9 time=0m18s total=24m31s
[ep_11100] ep=11001-11100 avg_reward=+354.0 avg_steps=100 epsilon=0.050 avg_loss=4.9 time=0m21s total=24m53s
[ep_11200] ep=11101-11200 avg_reward=+305.9 avg_steps=84 epsilon=0.050 avg_loss=4.9 time=0m18s total=25m11s
[ep_11300] ep=11201-11300 avg_reward=+304.6 avg_steps=87 epsilon=0.050 avg_loss=4.9 time=0m18s total=25m30s
[ep_11400] ep=11301-11400 avg_reward=+368.9 avg_steps=101 epsilon=0.050 avg_loss=4.8 time=0m21s total=25m52s
[ep_11500] ep=11401-11500 avg_reward=+342.0 avg_steps=93 epsilon=0.050 avg_loss=4.7 time=0m20s total=26m12s
[ep_11600] ep=11501-11600 avg_reward=+304.7 avg_steps=86 epsilon=0.050 avg_loss=4.7 time=0m18s total=26m30s
[ep_11700] ep=11601-11700 avg_reward=+354.6 avg_steps=99 epsilon=0.050 avg_loss=4.7 time=0m21s total=26m52s
[ep_11800] ep=11701-11800 avg_reward=+308.3 avg_steps=89 epsilon=0.050 avg_loss=4.7 time=0m19s total=27m11s
[ep_11900] ep=11801-11900 avg_reward=+321.1 avg_steps=93 epsilon=0.050 avg_loss=4.6 time=0m20s total=27m31s
[ep_12000] ep=11901-12000 avg_reward=+334.3 avg_steps=97 epsilon=0.050 avg_loss=4.7 time=0m20s total=27m52s
[ep_12100] ep=12001-12100 avg_reward=+367.0 avg_steps=104 epsilon=0.050 avg_loss=4.6 time=0m22s total=28m15s
[ep_12200] ep=12101-12200 avg_reward=+346.9 avg_steps=103 epsilon=0.050 avg_loss=4.4 time=0m22s total=28m37s
[ep_12300] ep=12201-12300 avg_reward=+333.0 avg_steps=96 epsilon=0.050 avg_loss=4.5 time=0m20s total=28m58s
[ep_12400] ep=12301-12400 avg_reward=+341.2 avg_steps=98 epsilon=0.050 avg_loss=4.3 time=0m21s total=29m19s
[ep_12500] ep=12401-12500 avg_reward=+342.6 avg_steps=97 epsilon=0.050 avg_loss=4.4 time=0m21s total=29m40s
[ep_12600] ep=12501-12600 avg_reward=+334.0 avg_steps=92 epsilon=0.050 avg_loss=4.3 time=0m19s total=30m00s
[ep_12700] ep=12601-12700 avg_reward=+332.6 avg_steps=92 epsilon=0.050 avg_loss=4.4 time=0m20s total=30m20s
[ep_12800] ep=12701-12800 avg_reward=+343.0 avg_steps=96 epsilon=0.050 avg_loss=4.4 time=0m20s total=30m41s
[ep_12900] ep=12801-12900 avg_reward=+344.9 avg_steps=101 epsilon=0.050 avg_loss=4.5 time=0m21s total=31m02s
[ep_13000] ep=12901-13000 avg_reward=+292.4 avg_steps=83 epsilon=0.050 avg_loss=4.5 time=0m17s total=31m20s
[ep_13100] ep=13001-13100 avg_reward=+323.1 avg_steps=89 epsilon=0.050 avg_loss=4.6 time=0m19s total=31m39s
[ep_13200] ep=13101-13200 avg_reward=+310.8 avg_steps=86 epsilon=0.050 avg_loss=4.4 time=0m18s total=31m58s
[ep_13300] ep=13201-13300 avg_reward=+308.8 avg_steps=85 epsilon=0.050 avg_loss=4.4 time=0m18s total=32m16s
[ep_13400] ep=13301-13400 avg_reward=+395.9 avg_steps=111 epsilon=0.050 avg_loss=4.4 time=0m23s total=32m40s
[ep_13500] ep=13401-13500 avg_reward=+395.5 avg_steps=111 epsilon=0.050 avg_loss=4.3 time=0m23s total=33m03s
[ep_13600] ep=13501-13600 avg_reward=+384.9 avg_steps=107 epsilon=0.050 avg_loss=4.2 time=0m23s total=33m26s
[ep_13700] ep=13601-13700 avg_reward=+342.1 avg_steps=96 epsilon=0.050 avg_loss=4.3 time=0m20s total=33m47s
[ep_13800] ep=13701-13800 avg_reward=+319.3 avg_steps=92 epsilon=0.050 avg_loss=4.3 time=0m21s total=34m09s
[ep_13900] ep=13801-13900 avg_reward=+337.6 avg_steps=98 epsilon=0.050 avg_loss=4.2 time=0m25s total=34m35s
[ep_14000] ep=13901-14000 avg_reward=+302.8 avg_steps=85 epsilon=0.050 avg_loss=4.3 time=0m18s total=34m53s
[ep_14100] ep=14001-14100 avg_reward=+337.7 avg_steps=94 epsilon=0.050 avg_loss=4.4 time=0m20s total=35m13s
[ep_14200] ep=14101-14200 avg_reward=+315.7 avg_steps=88 epsilon=0.050 avg_loss=4.2 time=0m18s total=35m32s
[ep_14300] ep=14201-14300 avg_reward=+337.8 avg_steps=99 epsilon=0.050 avg_loss=4.3 time=0m21s total=35m53s
[ep_14400] ep=14301-14400 avg_reward=+313.3 avg_steps=88 epsilon=0.050 avg_loss=4.3 time=0m18s total=36m12s
[ep_14500] ep=14401-14500 avg_reward=+322.5 avg_steps=91 epsilon=0.050 avg_loss=4.3 time=0m19s total=36m32s
[ep_14600] ep=14501-14600 avg_reward=+385.1 avg_steps=106 epsilon=0.050 avg_loss=4.2 time=0m22s total=36m55s
[ep_14700] ep=14601-14700 avg_reward=+333.2 avg_steps=92 epsilon=0.050 avg_loss=4.1 time=0m19s total=37m14s
[ep_14800] ep=14701-14800 avg_reward=+325.7 avg_steps=92 epsilon=0.050 avg_loss=4.0 time=0m19s total=37m34s
[ep_14900] ep=14801-14900 avg_reward=+308.1 avg_steps=87 epsilon=0.050 avg_loss=4.2 time=0m18s total=37m53s
[ep_15000] ep=14901-15000 avg_reward=+374.5 avg_steps=103 epsilon=0.050 avg_loss=4.2 time=0m22s total=38m15s
[ep_15100] ep=15001-15100 avg_reward=+334.0 avg_steps=94 epsilon=0.050 avg_loss=4.3 time=0m20s total=38m35s
[ep_15200] ep=15101-15200 avg_reward=+363.1 avg_steps=97 epsilon=0.050 avg_loss=4.4 time=0m20s total=38m56s
[ep_15300] ep=15201-15300 avg_reward=+367.9 avg_steps=104 epsilon=0.050 avg_loss=4.5 time=0m22s total=39m18s
[ep_15400] ep=15301-15400 avg_reward=+367.8 avg_steps=99 epsilon=0.050 avg_loss=4.4 time=0m21s total=39m39s
[ep_15500] ep=15401-15500 avg_reward=+331.6 avg_steps=93 epsilon=0.050 avg_loss=4.2 time=0m19s total=39m59s
[ep_15600] ep=15501-15600 avg_reward=+349.4 avg_steps=97 epsilon=0.050 avg_loss=4.1 time=0m20s total=40m20s
[ep_15700] ep=15601-15700 avg_reward=+374.1 avg_steps=108 epsilon=0.050 avg_loss=4.1 time=0m23s total=40m43s
[ep_15800] ep=15701-15800 avg_reward=+345.4 avg_steps=95 epsilon=0.050 avg_loss=4.1 time=0m20s total=41m04s
[ep_15900] ep=15801-15900 avg_reward=+372.3 avg_steps=108 epsilon=0.050 avg_loss=4.1 time=0m23s total=41m27s
[ep_16000] ep=15901-16000 avg_reward=+351.0 avg_steps=95 epsilon=0.050 avg_loss=4.2 time=0m20s total=41m47s
[ep_16100] ep=16001-16100 avg_reward=+345.8 avg_steps=99 epsilon=0.050 avg_loss=4.2 time=0m21s total=42m08s
[ep_16200] ep=16101-16200 avg_reward=+403.9 avg_steps=110 epsilon=0.050 avg_loss=4.2 time=0m23s total=42m32s
[ep_16300] ep=16201-16300 avg_reward=+424.1 avg_steps=116 epsilon=0.050 avg_loss=4.2 time=0m24s total=42m57s
[ep_16400] ep=16301-16400 avg_reward=+383.7 avg_steps=104 epsilon=0.050 avg_loss=4.3 time=0m22s total=43m19s
[ep_16500] ep=16401-16500 avg_reward=+356.4 avg_steps=97 epsilon=0.050 avg_loss=4.1 time=0m20s total=43m40s
[ep_16600] ep=16501-16600 avg_reward=+367.3 avg_steps=104 epsilon=0.050 avg_loss=4.1 time=0m22s total=44m02s
[ep_16700] ep=16601-16700 avg_reward=+322.2 avg_steps=96 epsilon=0.050 avg_loss=4.0 time=0m20s total=44m23s
[ep_16800] ep=16701-16800 avg_reward=+376.6 avg_steps=103 epsilon=0.050 avg_loss=4.1 time=0m22s total=44m45s
[ep_16900] ep=16801-16900 avg_reward=+340.6 avg_steps=96 epsilon=0.050 avg_loss=4.1 time=0m20s total=45m06s
[ep_17000] ep=16901-17000 avg_reward=+353.4 avg_steps=99 epsilon=0.050 avg_loss=4.0 time=0m21s total=45m27s
[ep_17100] ep=17001-17100 avg_reward=+304.6 avg_steps=87 epsilon=0.050 avg_loss=4.0 time=0m18s total=45m46s
[ep_17200] ep=17101-17200 avg_reward=+322.3 avg_steps=90 epsilon=0.050 avg_loss=3.9 time=0m19s total=46m05s
[ep_17300] ep=17201-17300 avg_reward=+376.1 avg_steps=105 epsilon=0.050 avg_loss=4.0 time=0m22s total=46m28s
[ep_17400] ep=17301-17400 avg_reward=+333.3 avg_steps=95 epsilon=0.050 avg_loss=4.1 time=0m20s total=46m48s
[ep_17500] ep=17401-17500 avg_reward=+363.4 avg_steps=104 epsilon=0.050 avg_loss=4.0 time=0m22s total=47m10s
[ep_17600] ep=17501-17600 avg_reward=+336.3 avg_steps=93 epsilon=0.050 avg_loss=3.9 time=0m20s total=47m30s
[ep_17700] ep=17601-17700 avg_reward=+347.2 avg_steps=95 epsilon=0.050 avg_loss=4.1 time=0m20s total=47m51s
[ep_17800] ep=17701-17800 avg_reward=+344.4 avg_steps=98 epsilon=0.050 avg_loss=4.0 time=0m21s total=48m12s
[ep_17900] ep=17801-17900 avg_reward=+371.6 avg_steps=109 epsilon=0.050 avg_loss=4.0 time=0m23s total=48m35s
[ep_18000] ep=17901-18000 avg_reward=+433.1 avg_steps=119 epsilon=0.050 avg_loss=4.1 time=0m25s total=49m00s
[ep_18100] ep=18001-18100 avg_reward=+317.4 avg_steps=90 epsilon=0.050 avg_loss=4.2 time=0m19s total=49m20s
[ep_18200] ep=18101-18200 avg_reward=+381.8 avg_steps=106 epsilon=0.050 avg_loss=4.2 time=0m22s total=49m42s
[ep_18300] ep=18201-18300 avg_reward=+377.0 avg_steps=105 epsilon=0.050 avg_loss=4.1 time=0m22s total=50m05s
[ep_18400] ep=18301-18400 avg_reward=+377.0 avg_steps=101 epsilon=0.050 avg_loss=4.0 time=0m21s total=50m26s
[ep_18500] ep=18401-18500 avg_reward=+356.1 avg_steps=101 epsilon=0.050 avg_loss=4.0 time=0m21s total=50m48s
[ep_18600] ep=18501-18600 avg_reward=+409.6 avg_steps=109 epsilon=0.050 avg_loss=4.1 time=0m23s total=51m11s
[ep_18700] ep=18601-18700 avg_reward=+338.2 avg_steps=94 epsilon=0.050 avg_loss=4.1 time=0m20s total=51m31s
[ep_18800] ep=18701-18800 avg_reward=+354.6 avg_steps=104 epsilon=0.050 avg_loss=4.2 time=0m22s total=51m53s
[ep_18900] ep=18801-18900 avg_reward=+333.9 avg_steps=94 epsilon=0.050 avg_loss=4.2 time=0m20s total=52m14s
[ep_19000] ep=18901-19000 avg_reward=+388.2 avg_steps=106 epsilon=0.050 avg_loss=4.4 time=0m22s total=52m36s
[ep_19100] ep=19001-19100 avg_reward=+368.2 avg_steps=100 epsilon=0.050 avg_loss=4.3 time=0m21s total=52m58s
[ep_19200] ep=19101-19200 avg_reward=+435.8 avg_steps=121 epsilon=0.050 avg_loss=4.2 time=0m25s total=53m24s
[ep_19300] ep=19201-19300 avg_reward=+345.8 avg_steps=99 epsilon=0.050 avg_loss=4.1 time=0m21s total=53m45s
[ep_19400] ep=19301-19400 avg_reward=+372.5 avg_steps=104 epsilon=0.050 avg_loss=4.1 time=0m22s total=54m07s
[ep_19500] ep=19401-19500 avg_reward=+374.1 avg_steps=103 epsilon=0.050 avg_loss=3.9 time=0m21s total=54m29s
[ep_19600] ep=19501-19600 avg_reward=+387.1 avg_steps=109 epsilon=0.050 avg_loss=3.9 time=0m23s total=54m53s
[ep_19700] ep=19601-19700 avg_reward=+360.1 avg_steps=99 epsilon=0.050 avg_loss=3.9 time=0m21s total=55m14s
[ep_19800] ep=19701-19800 avg_reward=+387.3 avg_steps=107 epsilon=0.050 avg_loss=4.0 time=0m22s total=55m37s
[ep_19900] ep=19801-19900 avg_reward=+363.4 avg_steps=98 epsilon=0.050 avg_loss=3.9 time=0m20s total=55m58s
[ep_20000] ep=19901-20000 avg_reward=+335.6 avg_steps=94 epsilon=0.050 avg_loss=4.0 time=0m19s total=56m17s