Imported from https://github.com/honda-keio/ppo-pytorch