[tune] Usability tweaks by richardliaw · Pull Request #2874 · ray-project/ray

richardliaw · 2018-09-14T06:51:31Z

What do these changes do?

Test

Related issue number

AmplabJenkins · 2018-09-14T07:34:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8202/
Test FAILed.

ericl

Shall we also add num iters to the summary line?

ericl · 2018-09-14T19:07:28Z

-                messages.append(" - {}:\t{}".format(t, t.progress_string()))
+            sorted_trials = sorted(trials, key=lambda t: t.experiment_tag)
            if len(trials) > limit:
+                tail_length = math.floor(limit/2)


Consider int div '//' instead

richardliaw · 2018-09-14T20:34:01Z

Isn't iter already shown?

Code:

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import ray
from ray.tune import run_experiments, register_trainable
from ray.tune.schedulers import AsyncHyperBandScheduler
from ray.tune.suggest import HyperOptSearch


def easy_objective(config, reporter):
    import time
    time.sleep(0.2)
    assert type(config["activation"]) == str, \
        "Config is incorrect: {}".format(type(config["activation"]))
    for i in range(100):
        reporter(
            timesteps_total=i,
            neg_mean_loss=-(config["height"] - 14)**2 +
            abs(config["width"] - 3))
        time.sleep(0.02)

if __name__ == '__main__':
    import argparse
    from hyperopt import hp

    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--smoke-test", action="store_true", help="Finish quickly for testing")
    args, _ = parser.parse_known_args()
    ray.init(redirect_output=True)

    register_trainable("exp", easy_objective)

    space = {
        'width': hp.uniform('width', 0, 20),
        'height': hp.uniform('height', -100, 100),
        'activation': hp.choice("activation", ["relu", "tanh"])
    }

    config = {
        "my_exp": {
            "run": "exp",
            "num_samples": 10 if args.smoke_test else 1000,
            "stop": {
                "training_iteration": 100
            },
        }
    }
    algo = HyperOptSearch(space, max_concurrent=4, reward_attr="neg_mean_loss")
    scheduler = AsyncHyperBandScheduler(reward_attr="neg_mean_loss")
    run_experiments(config, search_alg=algo, scheduler=scheduler)

gives me output:

== Status ==
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 90.000: None | Iter 30.000: None | Iter 10.000: None
Bracket: Iter 90.000: None | Iter 30.000: None
Bracket: Iter 90.000: None
Resources requested: 4/48 CPUs, 0/1 GPUs
Result logdir: /home/eecs/rliaw/ray_results/my_exp
RUNNING trials:
 - exp_10_n=tanh,t=-17.95,h=8.5007:	RUNNING
 - exp_11_n=tanh,t=-37.719,h=8.6749:	RUNNING
 - exp_12_n=relu,t=-17.59,h=5.3634:	RUNNING
 - exp_9_n=tanh,t=58.015,h=6.2248:	RUNNING [pid=27364], 1 s, 1 iter, 39 ts
TERMINATED trials:
 - exp_1_n=relu,t=-16.999,h=13.989:	TERMINATED [pid=23841], 4 s, 4 iter, 99 ts
 - exp_2_n=tanh,t=-15.736,h=8.2097:	TERMINATED [pid=23842], 4 s, 4 iter, 99 ts
 - exp_3_n=relu,t=71.59,h=9.7102:	TERMINATED [pid=23830], 4 s, 4 iter, 99 ts
 - exp_4_n=tanh,t=-27.854,h=9.1752:	TERMINATED [pid=23824], 4 s, 4 iter, 99 ts
 - exp_5_n=tanh,t=-58.677,h=9.5805:	TERMINATED [pid=23811], 4 s, 4 iter, 99 ts
 - exp_6_n=tanh,t=64.094,h=1.6311:	TERMINATED [pid=23839], 4 s, 4 iter, 99 ts
 - exp_7_n=tanh,t=76.081,h=18.001:	TERMINATED [pid=23844], 4 s, 4 iter, 99 ts
 - exp_8_n=tanh,t=-29.688,h=4.1828:	TERMINATED [pid=27191], 4 s, 4 iter, 99 ts

on this branch right now.

ericl · 2018-09-14T20:37:45Z

Hm weird, maybe it was something else.

…

On Fri, Sep 14, 2018, 1:34 PM Richard Liaw ***@***.***> wrote: Isn't iter already shown? Code: from __future__ import absolute_importfrom __future__ import divisionfrom __future__ import print_function import rayfrom ray.tune import run_experiments, register_trainablefrom ray.tune.schedulers import AsyncHyperBandSchedulerfrom ray.tune.suggest import HyperOptSearch def easy_objective(config, reporter): import time time.sleep(0.2) assert type(config["activation"]) == str, \ "Config is incorrect: {}".format(type(config["activation"])) for i in range(100): reporter( timesteps_total=i, neg_mean_loss=-(config["height"] - 14)**2 + abs(config["width"] - 3)) time.sleep(0.02) if __name__ == '__main__': import argparse from hyperopt import hp parser = argparse.ArgumentParser() parser.add_argument( "--smoke-test", action="store_true", help="Finish quickly for testing") args, _ = parser.parse_known_args() ray.init(redirect_output=True) register_trainable("exp", easy_objective) space = { 'width': hp.uniform('width', 0, 20), 'height': hp.uniform('height', -100, 100), 'activation': hp.choice("activation", ["relu", "tanh"]) } config = { "my_exp": { "run": "exp", "num_samples": 10 if args.smoke_test else 1000, "stop": { "training_iteration": 100 }, } } algo = HyperOptSearch(space, max_concurrent=4, reward_attr="neg_mean_loss") scheduler = AsyncHyperBandScheduler(reward_attr="neg_mean_loss") run_experiments(config, search_alg=algo, scheduler=scheduler) gives me output: == Status == Using AsyncHyperBand: num_stopped=0 Bracket: Iter 90.000: None | Iter 30.000: None | Iter 10.000: None Bracket: Iter 90.000: None | Iter 30.000: None Bracket: Iter 90.000: None Resources requested: 4/48 CPUs, 0/1 GPUs Result logdir: /home/eecs/rliaw/ray_results/my_exp RUNNING trials: - exp_10_n=tanh,t=-17.95,h=8.5007: RUNNING - exp_11_n=tanh,t=-37.719,h=8.6749: RUNNING - exp_12_n=relu,t=-17.59,h=5.3634: RUNNING - exp_9_n=tanh,t=58.015,h=6.2248: RUNNING [pid=27364], 1 s, 1 iter, 39 ts TERMINATED trials: - exp_1_n=relu,t=-16.999,h=13.989: TERMINATED [pid=23841], 4 s, 4 iter, 99 ts - exp_2_n=tanh,t=-15.736,h=8.2097: TERMINATED [pid=23842], 4 s, 4 iter, 99 ts - exp_3_n=relu,t=71.59,h=9.7102: TERMINATED [pid=23830], 4 s, 4 iter, 99 ts - exp_4_n=tanh,t=-27.854,h=9.1752: TERMINATED [pid=23824], 4 s, 4 iter, 99 ts - exp_5_n=tanh,t=-58.677,h=9.5805: TERMINATED [pid=23811], 4 s, 4 iter, 99 ts - exp_6_n=tanh,t=64.094,h=1.6311: TERMINATED [pid=23839], 4 s, 4 iter, 99 ts - exp_7_n=tanh,t=76.081,h=18.001: TERMINATED [pid=23844], 4 s, 4 iter, 99 ts - exp_8_n=tanh,t=-29.688,h=4.1828: TERMINATED [pid=27191], 4 s, 4 iter, 99 ts on this branch right now. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2874 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA6Sthx4m5INTfeArjqWVq6wsJpGQl0ks5ubBLFgaJpZM4Wotj9> .

richardliaw · 2018-09-14T20:48:26Z

OK cool.

Right now the TODO is that trials need to be sorted numerically rather than by string...

richardliaw · 2018-09-15T03:59:44Z

Should now look like:

== Status ==
Using AsyncHyperBand: num_stopped=0
Bracket: Iter 90.000: None | Iter 30.000: None | Iter 10.000: None
Bracket: Iter 90.000: None | Iter 30.000: None
Bracket: Iter 90.000: None
Resources requested: 40/48 CPUs, 0/1 GPUs
Result logdir: /home/eecs/rliaw/ray_results/my_exp
RUNNING trials:
 - exp_23_n=relu,t=69.882,h=3.8984:     RUNNING [pid=44386], 2 s, 2 iter, 91 ts
 - exp_30_n=relu,t=34.484,h=1.941:      RUNNING [pid=44395], 1 s, 1 iter, 39 ts
 - exp_42_n=tanh,t=-32.883,h=19.883:    RUNNING [pid=44392], 2 s, 2 iter, 99 ts
 - exp_48_n=tanh,t=28.183,h=10.711:     RUNNING
 - exp_52_n=tanh,t=-53.695,h=5.1436:    RUNNING
  ... 30 not shown
 - exp_107_n=tanh,t=48.245,h=11.913:    RUNNING
 - exp_108_n=tanh,t=1.1604,h=8.3018:    RUNNING
 - exp_109_n=relu,t=-7.9271,h=5.67:     RUNNING
 - exp_110_n=tanh,t=26.672,h=7.5366:    RUNNING
 - exp_111_n=tanh,t=39.678,h=4.0774:    RUNNING
TERMINATED trials:
 - exp_1_n=relu,t=25.078,h=6.932:       TERMINATED [pid=44376], 4 s, 4 iter, 99 ts
 - exp_2_n=tanh,t=-25.394,h=19.585:     TERMINATED [pid=44383], 4 s, 4 iter, 99 ts
 - exp_3_n=tanh,t=12.7,h=7.2313:        TERMINATED [pid=44387], 4 s, 4 iter, 99 ts
 - exp_4_n=relu,t=49.95,h=18.697:       TERMINATED [pid=44367], 4 s, 4 iter, 99 ts
 - exp_5_n=tanh,t=88.673,h=19.928:      TERMINATED [pid=44397], 4 s, 4 iter, 99 ts
  ... 61 not shown
 - exp_75_n=tanh,t=-4.6491,h=10.742:    TERMINATED [pid=695], 3 s, 3 iter, 99 ts
 - exp_76_n=tanh,t=-40.07,h=8.6659:     TERMINATED [pid=749], 3 s, 3 iter, 99 ts
 - exp_77_n=tanh,t=46.373,h=8.0366:     TERMINATED [pid=826], 3 s, 3 iter, 99 ts
 - exp_78_n=tanh,t=-29.942,h=9.8078:    TERMINATED [pid=908], 3 s, 3 iter, 99 ts
 - exp_80_n=relu,t=-0.78428,h=16.386:   TERMINATED [pid=1031], 3 s, 3 iter, 99 ts

ericl · 2018-09-15T04:26:00Z

+def _naturalize(string):
+    """Provides a natural representation for string for nice sorting."""
+    splits = re.split('([0-9]+)', string)
+    return [int(text) if text.isdigit() else text.lower() for text in splits]


Double quotes

AmplabJenkins · 2018-09-15T05:22:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8220/
Test PASSed.

AmplabJenkins · 2018-09-15T06:30:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8226/
Test PASSed.

richardliaw added 2 commits September 13, 2018 23:49

tweaks

d96287a

nit'

95fd1f4

ericl reviewed Sep 14, 2018

View reviewed changes

richardliaw added 2 commits September 14, 2018 20:56

Fix sorting for trials

45ae8d9

lint

588d18b

richardliaw assigned ericl Sep 15, 2018

ericl approved these changes Sep 15, 2018

View reviewed changes

double quotes

afd8a52

richardliaw merged commit e05baed into ray-project:master Sep 15, 2018

richardliaw deleted the tune_minor_usage branch September 15, 2018 18:02

richardliaw mentioned this pull request Oct 4, 2018

[tune] Tweaks to Trainable and Verbosity #2889

Merged

8 tasks

richardliaw mentioned this pull request Oct 26, 2018

[tune] No incrementing key when using hyperopt #2872

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[tune] Usability tweaks#2874

[tune] Usability tweaks#2874
richardliaw merged 5 commits into
ray-project:masterfrom
richardliaw:tune_minor_usage

richardliaw commented Sep 14, 2018 •

edited

Loading

Uh oh!

AmplabJenkins commented Sep 14, 2018

Uh oh!

ericl left a comment

Uh oh!

ericl Sep 14, 2018

Uh oh!

richardliaw commented Sep 14, 2018

Uh oh!

ericl commented Sep 14, 2018 via email

Uh oh!

richardliaw commented Sep 14, 2018 •

edited

Loading

Uh oh!

richardliaw commented Sep 15, 2018

Uh oh!

ericl Sep 15, 2018

Uh oh!

AmplabJenkins commented Sep 15, 2018

Uh oh!

AmplabJenkins commented Sep 15, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

richardliaw commented Sep 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Related issue number

Uh oh!

AmplabJenkins commented Sep 14, 2018

Uh oh!

ericl left a comment

Choose a reason for hiding this comment

Uh oh!

ericl Sep 14, 2018

Choose a reason for hiding this comment

Uh oh!

richardliaw commented Sep 14, 2018

Uh oh!

ericl commented Sep 14, 2018 via email

Uh oh!

richardliaw commented Sep 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

richardliaw commented Sep 15, 2018

Uh oh!

ericl Sep 15, 2018

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Sep 15, 2018

Uh oh!

AmplabJenkins commented Sep 15, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

richardliaw commented Sep 14, 2018 •

edited

Loading

richardliaw commented Sep 14, 2018 •

edited

Loading