ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedback

Correcting robot manipulation online without retraining the base policy

Muhammad A. Muttaqien*, Tomohiro Motoda*, Ryo Hanai, Yukiyasu Domae

Embodied AI Research Team · National Institute of AIST · Tokyo, Japan
* Equal contribution

Abstract

Robotic manipulation policies trained via imitation learning, such as Action Chunking with Transformers (ACT), can achieve strong performance under ideal conditions but often remain sensitive to small execution errors and distribution shifts. Correcting these failures typically requires dataset aggregation and full-policy retraining, which is computationally expensive and unsuitable for real-time deployment. In this work, we propose Online Residual Policy Adaptation (ORPA), a framework that enables immediate, feedback-driven correction of robot actions without modifying the underlying policy parameters. ORPA augments a pretrained control policy with a lightweight, feedback-conditioned module that predicts residual adjustments directly in joint space, allowing the system to adapt its behavior at runtime. We evaluate ORPA on a set of precision-sensitive manipulation tasks using the ALOHA platform, demonstrating improvements in success rate and recovery from small perturbations compared to baseline control policies and rule-based inverse kinematics corrections.

Why ORPA?

A strong manipulation policy does not always need to be retrained when it fails. Many failures arise from small local execution errors that only require a small corrective adjustment.

Full Retraining Requires additional demonstrations and repeated optimization.

Rule-based IK Uses predefined geometric corrections without sufficient task context.

          
          ORPA
          
            Learns context-dependent residual corrections during execution.

Method

ORPA keeps the pretrained ACT policy frozen and learns only the correction required for the current execution context. Human feedback is encoded and used to refine the action trajectory online.

Architecture of the proposed Online Residual Policy Adaptation framework

ORPA augments the frozen ACT policy with a lightweight feedback-conditioned module. Given the current execution context and human feedback, it predicts a residual correction that refines the original ACT action without modifying the base policy.

a_final = a_ACT + Δa_ORPA

Pretrained ACT Visual observations + robot states

Human Feedback Corrective instruction

Feedback Encoder Feedback representation

Policy Updater Predict residual Δa

Corrected Action ACT action + residual

Real-World Task Scores

0.83 0.86

Snack Box Transfer
ACT → ORPA

0.79 0.82

Chip Tube Opening
ACT → ORPA

0.80 0.83

Object Sorting
ACT → ORPA

Scores compare Original ACT with ORPA using action and observation perturbations.

Takeaway

What if a robot could correct a manipulation mistake immediately instead of retraining the entire policy? ORPA treats deployment failures as a residual adaptation problem. The pretrained ACT policy preserves its learned manipulation behavior, while human feedback provides the information needed to make small, context-dependent corrections directly in joint space. This enables improved recovery from execution perturbations with minimal additional computation and without modifying the base policy.

🚀 As for future work, ORPA could be extended toward richer language feedback, contact-aware corrections, broader manipulation skills, and integration with predictive world models (such as DreamerV4), to enable more context-aware, anticipatory, and adaptive robot behavior.

BibTeX

@article{muttaqien2026orpa,
  author    = {Muhammad A. Muttaqien and Tomohiro Motoda and Ryo Hanai and Yukiyasu Domae},
  title     = {ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedback},
  journal   = {arXiv preprint arXiv:2608.17323},
  year      = {2026},
  doi       = {10.48550/arXiv.2608.17323}
}