Skip to main content

Ergodicity-Aware Reinforcement Learning Under Deployment Shift

Modernized reinforcement-learning study comparing standard, log, and learned ergodicity-aware PPO, followed by frozen and adaptive policy evaluation under a hidden deployment-time dynamics shift.

Comparison of long-run log growth and median final wealth for standard and ergodicity-aware reinforcement-learning policies

This project studies how the reinforcement-learning objective changes long-run behavior in a non-ergodic multiplicative process. I implemented standard PPO, log-growth PPO, and a PPO agent using a learned ergodicity transformation, then extended the experiment to a hidden deployment-time change in the environment.

Baseline result

The ergodicity-aware agents learn substantially less aggressive allocation policies than standard PPO and achieve positive long-run log growth. The learned transformation recovers behavior close to the analytical log-optimal solution without being explicitly given the logarithm.

PolicyMedian final wealthMean log growthMean allocation
Standard PPO2.49-0.034140.809
Log PPO218.430.005050.210
Learned-transform PPO209.640.005830.224
Analytical optimum186.100.006230.250

Table 1. Ergodicity-aware objectives trade extreme ensemble gains for substantially stronger typical long-run performance.

Baseline comparison of mean log growth and median final wealth
Figure 1. Standard PPO is aggressive but performs poorly on typical trajectories; log and learned-transform PPO shift behavior toward positive long-run growth.

Deployment shift and adaptation

At step 50, the hidden favorable-outcome probability changes from 0.50 to 0.47, moving the analytical log-optimal allocation from 0.25 to 0.115. Frozen ergodicity-aware policies remain robust but do not fully adapt.

A naive rolling estimator detects the change but overreacts to noisy probability estimates. A Beta-Bernoulli regularized estimator stabilizes the controller, improving mean log growth from -0.00623 to 0.00153, although it adapts more slowly than the regime-aware oracle.

Comparison of frozen, naive adaptive, regularized adaptive, and oracle policies
Figure 2. Naive online adaptation overreacts to noisy estimates, while regularization restores stability but remains more conservative than the oracle.

Implementation

  • Python, Gymnasium, Stable-Baselines3, PyTorch
  • PPO with normalized continuous actions
  • LOWESS-based learned ergodicity transformation
  • Fast interpolated reward transformation for training
  • Common-random-number Monte Carlo evaluation
  • Rolling and Beta-Bernoulli online adaptation

This is a compact proof-of-concept rather than a full continual-RL method. The next step is a multi-seed study and extension to a richer dynamical-control environment.