Ensure AutoML respects max_runtime_secs

Description

AutoML takes longer time than the time given. Sometimes 2.7-2.9 times.

A couple of examples from Jenkins run 6666
http://mr-0xc1:8080/view/H2OAI/job/h2oai-benchmark-quick/1229/consoleFull

AutoML given same time as H2OAI but takes much longer

H2OAI time (sec): 253.00870013237
22:56:39 AutoML original time (sec): 685.0196287631989
2.7x

23:17:21 H2OAI time (sec): 288.0384900569916
23:17:21 AutoML original time (sec): 841.4690053462982
2.9x

etc

Activity

Show:
Magnus Stensmo
July 10, 2018, 3:07 PM
Edited

New record set:

pc_letter test set metrics (logloss):
H2O original : 0.1339664666266977
H2OAI munged: 0.12747641588240172
H2OAI time (sec): 8695.321388721466
AutoML original time (sec): 59342.67062044144
AutoML slower than h2oai by 50647.349 s = 582.47%

That’s 16.5 hours when given 2.4 hours — 6.8 times longer

http://mr-0xc1:8080/job/h2oai-benchmark-many-defaults/19/consoleFull

Erin LeDell
July 12, 2018, 6:48 AM

A PR was made, but it was buggy so we had to revert it. Check the PR for code you can use.

Assignee

Sebastien Poirier

Fix versions

Reporter

Magnus Stensmo

Support ticket URL

None

Labels

None

Affected Spark version

None

Customer Request Type

None

Task progress

None

ReleaseNotesHidden

None

CustomerVisible

No

Epic Link

Components

Priority

Major
Configure