[rllib] example and docs on how to use parametric actions with DQN / PG algorithms by ericl · Pull Request #3384 · ray-project/ray

ericl · 2018-11-22T07:24:07Z

What do these changes do?

Add examples of how to work with parametric action spaces (e.g., OpenAI 5 style).

Related issue number

ericl · 2018-11-22T07:24:59Z


    def transform(self, observation):
+        if not isinstance(observation, OrderedDict):
+            observation = OrderedDict(sorted(list(observation.items())))


oops this is kind of an important bug fix

do you need to match the rest of the checks? https://github.com/openai/gym/blob/master/gym/spaces/dict_space.py#L34

No, note that that is a check against the space spec, this is sorting the observation dict.

AmplabJenkins · 2018-11-22T09:05:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9538/
Test FAILed.

AmplabJenkins · 2018-11-22T10:27:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9536/
Test FAILed.

AmplabJenkins · 2018-11-22T10:53:14Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9539/
Test FAILed.

AmplabJenkins · 2018-11-22T10:58:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9540/
Test FAILed.

AmplabJenkins · 2018-11-22T11:51:32Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9541/
Test FAILed.

AmplabJenkins · 2018-11-22T13:22:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9544/
Test FAILed.

AmplabJenkins · 2018-11-22T20:41:15Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9546/
Test FAILed.

AmplabJenkins · 2018-11-22T22:41:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9548/
Test FAILed.

AmplabJenkins · 2018-11-22T22:51:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9549/
Test FAILed.

AmplabJenkins · 2018-11-22T23:11:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9550/
Test FAILed.

AmplabJenkins · 2018-11-22T23:21:57Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9551/
Test FAILed.

AmplabJenkins · 2018-11-22T23:46:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9552/
Test FAILed.

AmplabJenkins · 2018-11-23T00:46:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9554/
Test FAILed.

AmplabJenkins · 2018-11-23T01:02:00Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9555/
Test FAILed.

AmplabJenkins · 2018-11-23T01:11:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9556/
Test FAILed.

zegerhoogeboom · 2018-11-23T13:43:56Z

Very sorry for my wrong previous comment, the issue was simply that I didn't install Ray from your branch. Both DQN and the PPO are working beautifully with the masking. I'm not using the action embeddings, but those are at least working in the example you created.

ericl · 2018-11-23T17:34:51Z

That should be fixed in the latest update -- try pulling.

AmplabJenkins · 2018-11-25T03:31:14Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9576/
Test FAILed.

ericl · 2018-11-27T09:12:24Z

Ping @richardliaw

richardliaw

tf.boolean_mask(x, tf.logical_not(tf.is_inf(x))) could be cleaner (for future ref)

richardliaw · 2018-11-24T08:04:28Z


    def transform(self, observation):
+        if not isinstance(observation, OrderedDict):
+            observation = OrderedDict(sorted(list(observation.items())))


do you need to match the rest of the checks? https://github.com/openai/gym/blob/master/gym/spaces/dict_space.py#L34

ericl

The reason is_inf won't work is because inf is numerically unstable, so we use tf.float32.min instead

ericl added 3 commits November 21, 2018 22:27

pa model

95b2aa0

docs

2c3ca54

lint

9633107

ericl assigned richardliaw Nov 22, 2018

ericl commented Nov 22, 2018

View reviewed changes

ericl added 2 commits November 21, 2018 23:47

update

a95f0db

doc

caca39a

ericl force-pushed the pa-model branch from 4d86ce5 to caca39a Compare November 22, 2018 07:49

large

de5cd1b

ericl force-pushed the pa-model branch from e34e117 to de5cd1b Compare November 22, 2018 07:58

Update rllib-training.rst

26dcc8f

ericl added 5 commits November 22, 2018 09:40

Update parametric_action_cartpole.py

fd2058d

doc dqn

0a50f72

doc

3877d6a

Merge branch 'pa-model' of github.com:ericl/ray into pa-model

b1dd888

stop at 50

85d9fb8

ericl changed the title ~~[rllib] example and docs on how to use parametric actions with pg algorithms~~ [rllib] example and docs on how to use parametric actions with DQN / PG algorithms Nov 22, 2018

ericl added 4 commits November 22, 2018 11:49

0.2 gpu

6187de0

masked

b707bd7

improve docs

ee79f73

gym env

90b52e1

ericl added 2 commits November 22, 2018 12:43

custom lstm doc

084e8c7

token

115977c

ericl added 2 commits November 22, 2018 13:42

Merge remote-tracking branch 'upstream/master' into pa-model

1ca2992

support dueling

78c765c

ericl force-pushed the pa-model branch from e83ebde to 78c765c Compare November 22, 2018 22:00

Update rllib-models.rst

06148b4

ericl added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Nov 23, 2018

128

02b886d

richardliaw approved these changes Nov 28, 2018

View reviewed changes

ericl commented Nov 28, 2018

View reviewed changes

ericl merged commit f0df97d into ray-project:master Nov 28, 2018

Uh oh!

Conversation

ericl commented Nov 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Related issue number

Uh oh!

ericl Nov 22, 2018

Choose a reason for hiding this comment

Uh oh!

richardliaw Nov 24, 2018

Choose a reason for hiding this comment

Uh oh!

ericl Nov 28, 2018

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 23, 2018

Uh oh!

AmplabJenkins commented Nov 23, 2018

Uh oh!

AmplabJenkins commented Nov 23, 2018

Uh oh!

zegerhoogeboom commented Nov 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericl commented Nov 23, 2018

Uh oh!

AmplabJenkins commented Nov 25, 2018

Uh oh!

ericl commented Nov 27, 2018

Uh oh!

richardliaw left a comment

Choose a reason for hiding this comment

Uh oh!

richardliaw Nov 24, 2018

Choose a reason for hiding this comment

Uh oh!

ericl left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ericl commented Nov 22, 2018 •

edited

Loading

zegerhoogeboom commented Nov 23, 2018 •

edited

Loading