MURM: Utilization of Multi-Views for Goal-Conditioned Reinforcement Learning in Robotic Manipulation

Seongwon Jang, Hyemi Jeong, Hyunseok Yang

Research output: Contribution to journalArticlepeer-review

Abstract

We present a novel framework, multi-view unified reinforcement learning for robotic manipulation (MURM), which efficiently utilizes multiple camera views to train a goal-conditioned policy for a robot to perform complex tasks. The MURM framework consists of three main phases: (i) demo collection from an expert, (ii) representation learning, and (iii) offline reinforcement learning. In the demo collection phase, we design a scripted expert policy that uses privileged information, such as Cartesian coordinates of a target and goal, to solve the tasks. We add noise to the expert policy to provide sufficient interactive information about the environment, as well as suboptimal behavioral trajectories. We designed three tasks in a Pybullet simulation environment, including placing an object in a desired goal position and picking up various objects that are randomly positioned in the environment. In the representation learning phase, we use a vector-quantized variational autoencoder (VQVAE) to learn a more structured latent representation that makes it feasible to train for RL compared to high-dimensional raw images. We train VQVAE models for each distinct camera view and define the best viewpoint settings for training. In the offline reinforcement learning phase, we use the Implicit Q-learning (IQL) algorithm as our baseline and introduce a separated Q-functions method and dropout method that can be implemented in multi-view settings to train the goal-conditioned policy with supervised goal images. We conduct experiments in simulation and show that the single-view baseline fails to solve complex tasks, whereas MURM is successful.

Original languageEnglish
Article number119
JournalRobotics
Volume12
Issue number4
DOIs
Publication statusPublished - 2023 Aug

Bibliographical note

Publisher Copyright:
© 2023 by the authors.

All Science Journal Classification (ASJC) codes

  • Mechanical Engineering
  • Control and Optimization
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'MURM: Utilization of Multi-Views for Goal-Conditioned Reinforcement Learning in Robotic Manipulation'. Together they form a unique fingerprint.

Cite this