Robotics: Science and Systems XIV

Simplifying Reward Design through Divide-and-Conquer

Ellis Ratner, Dylan Hadfield-Menell, Anca Dragan

Abstract:

Designing a good reward function is essential to robot planning and reinforcement learning, however it can be both challenging and frustrating. The reward needs to work across multiple different environments, and that often requires many iterations of tuning. We introduce a novel divide-and-conquer approach that enables the designer to specify a reward separately for each environment. By treating these separate reward functions as observations about the underlying true reward, we derive an approach to infer a common reward across all environments. We conduct user studies in an abstract grid world domain and a motion planning domain for a 7-DOF manipulator that measure user effort and solution quality. We show that our method is faster, easier to use, and produces a higher quality solution than the typical method of designing a reward jointly across all environments. We additionally conduct a series of experiments that measure the sensitivity of these results to different properties of the reward design task such as number of environments, the number of feasible solutions per environment, and the fraction of the total features that vary within each environment. We find that independent reward design compares favorably with the standard, joint, reward design process but works best when the design problem can be divided into simpler subproblems.

Download:

Bibtex:

  
@INPROCEEDINGS{Ratner-RSS-18, 
    AUTHOR    = {Ellis Ratner AND Dylan Hadfield-Menell AND Anca Dragan}, 
    TITLE     = {Simplifying Reward Design through Divide-and-Conquer}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2018}, 
    ADDRESS   = {Pittsburgh, Pennsylvania}, 
    MONTH     = {June}, 
    DOI       = {10.15607/RSS.2018.XIV.048} 
}