Why Bounded Residual Actions Are Not a Safety Guarantee in Robot Learning

Nominal controller and bounded residual RL policy followed by a state-dependent safety filter — Bounding the residual limits the learner’s authority. A safety filter addresses a different question: whether the candidate action is acceptable for the current state.

If a reinforcement-learning policy can only apply small corrections, how dangerous can it really be?

This sounds reasonable. If the learned controller cannot generate arbitrarily large torques or forces, then at least its authority is limited.

That is useful. But it is not a safety guarantee.

The important distinction is simple:

Bounding what the learner can command is not the same as bounding where the closed-loop system can go.

Bounded action is not bounded state

Consider residual control:

u = u_{\mathrm{nom}} + u_{\mathrm{RL}},

with a bounded learned correction

\lVert u_{\mathrm{RL}} \rVert \leq \bar{u}.

This limits the authority of the RL policy.

But a safety condition usually concerns the state:

x(t) \in \mathcal{X}_{\mathrm{safe}}.

The first statement does not imply the second. A small input can still move a dynamical system toward a joint limit, collision region, unstable configuration, or another unsafe state.

There is also a second detail: bounding only $u_{\mathrm{RL}}$ does not automatically guarantee that the total command $u$ respects actuator limits. That must be checked separately.

A tiny counterexample

Take

x_{k+1}=x_k+\Delta t\,u_k,

with

|u_k|\leq0.5, \qquad x_k\leq1.

Let $\Delta t=0.1$ and $x_0=0.8$ . If the controller repeatedly applies $u_k=0.5$ , every command respects the action bound, but

x_k=0.8+0.05k.

After five steps,

x_5=1.05>1.

Nothing extreme happened. Every action was bounded. The state constraint was still violated.

The bound controlled how fast the state moved, not where it was allowed to end up.

Why robots make this harder

Real robots add dynamics, geometry, contacts, uncertainty, and coupled constraints.

A bounded torque does not by itself prevent a joint-limit violation. A bounded velocity correction does not guarantee collision avoidance. In a cable-driven robot, even a moderate generalized torque may correspond to infeasible cable tensions at a particular configuration.

So a statement like

“the RL action is small”

is useful engineering information, but it is not enough to establish closed-loop safety.

The missing layer: state-dependent safety

A stronger architecture is

u_{\mathrm{candidate}}=u_{\mathrm{nom}}+u_{\mathrm{RL}},

followed by

u_{\mathrm{safe}}=\mathcal{F}(x,u_{\mathrm{candidate}}).

Here the safety mechanism sees the current state and evaluates the candidate command against system constraints.

This idea appears in several modern directions: Control Barrier Functions, predictive safety filters, shielded RL, and other constrained learning-control methods.

They differ mathematically, but the common point is important: safety reasoning depends on the state, dynamics, and constraints, not only on action magnitude.

This is also why current work on safe learning-based control is interesting. The difficult problem is not simply making the policy weaker. It is deciding when a learned action is admissible without throwing away the performance benefit of learning.

Connection to my own work

In my M.Sc. research, I used Computed Torque Control as the nominal controller for a cable-driven lower-limb rehabilitation robot and added a bounded residual DDPG policy to compensate for disturbances and parametric uncertainty.

Bounding the residual was a useful design choice because it restricted the learned component’s authority. I also checked joint motion and cable-tension feasibility in simulation.

But I do not interpret those choices as a proof of safety or stability. The work did not establish forward invariance of a safe set or use a formal CBF/MPC safety filter.

For me, that limitation points to a more interesting research question: how can learning improve performance while explicit safety machinery protects the closed-loop system?

Takeaway

Action bounds answer:

How much can the learner command?

Safety analysis asks:

Which closed-loop trajectories remain admissible?

Those questions are related, but they are not the same.

A bounded residual policy can be a sensible engineering choice. A safety guarantee requires something more.