Natural Value Approximators: Learning when to Trust Past Estimates

lib:4d89829e130646b7 (v1.0.0)

Authors: Zhongwen Xu,Joseph Modayil,Hado P. Van Hasselt,Andre Barreto,David Silver,Tom Schaul
Where published: NeurIPS 2017 12
Document:  PDF  DOI 
Abstract URL: http://papers.nips.cc/paper/6807-natural-value-approximators-learning-when-to-trust-past-estimates


Neural networks have a smooth initial inductive bias, such that small changes in input do not lead to large changes in output. However, in reinforcement learning domains with sparse rewards, value functions have non-smooth structure with a characteristic asymmetric discontinuity whenever rewards arrive. We propose a mechanism that learns an interpolation between a direct value estimate and a projected value estimate computed from the encountered reward and the previous estimate. This reduces the need to learn about discontinuities, and thus improves the value function approximation. Furthermore, as the interpolation is learned and state-dependent, our method can deal with heterogeneous observability. We demonstrate that this one change leads to significant improvements on multiple Atari games, when applied to the state-of-the-art A3C algorithm.

Relevant initiatives  

Related knowledge about this paper Reproduced results (crowd-benchmarking and competitions) Artifact and reproducibility checklists Common formats for research projects and shared artifacts Reproducibility initiatives

Comments  

Please log in to add your comments!
If you notice any inapropriate content that should not be here, please report us as soon as possible and we will try to remove it within 48 hours!