Ergodicity-Aware Reinforcement Learning Under Deployment Shift
Modernized reinforcement-learning study comparing standard, log, and learned ergodicity-aware PPO, followed by frozen and adaptive policy evaluation under a hidden deployment-time dynamics shift.
This project studies how the reinforcement-learning objective changes long-run behavior in a non-ergodic multiplicative process. I implemented standard PPO, log-growth PPO, and a PPO agent using a learned ergodicity transformation, then extended the experiment to a hidden deployment-time change in the environment.
View the reproducible Python project on GitHub
Baseline result
The ergodicity-aware agents learn substantially less aggressive allocation policies than standard PPO and achieve positive long-run log growth. The learned transformation recovers behavior close to the analytical log-optimal solution without being explicitly given the logarithm.
| Policy | Median final wealth | Mean log growth | Mean allocation |
|---|---|---|---|
| Standard PPO | 2.49 | -0.03414 | 0.809 |
| Log PPO | 218.43 | 0.00505 | 0.210 |
| Learned-transform PPO | 209.64 | 0.00583 | 0.224 |
| Analytical optimum | 186.10 | 0.00623 | 0.250 |
Table 1. Ergodicity-aware objectives trade extreme ensemble gains for substantially stronger typical long-run performance.
Deployment shift and adaptation
At step 50, the hidden favorable-outcome probability changes from 0.50 to 0.47, moving the analytical log-optimal allocation from 0.25 to 0.115. Frozen ergodicity-aware policies remain robust but do not fully adapt.
A naive rolling estimator detects the change but overreacts to noisy probability estimates. A Beta-Bernoulli regularized estimator stabilizes the controller, improving mean log growth from -0.00623 to 0.00153, although it adapts more slowly than the regime-aware oracle.
Implementation
- Python, Gymnasium, Stable-Baselines3, PyTorch
- PPO with normalized continuous actions
- LOWESS-based learned ergodicity transformation
- Fast interpolated reward transformation for training
- Common-random-number Monte Carlo evaluation
- Rolling and Beta-Bernoulli online adaptation
This is a compact proof-of-concept rather than a full continual-RL method. The next step is a multi-seed study and extension to a richer dynamical-control environment.