Unable to run multiple H2O AutoML processes simultaneously

Description

As a machine learning engineer, I need to be able to run multiple H2o AutoML processes simultaneously. I get the below error when attempting to run multiple instances on the same H2o server – but this should be possible. I'm wondering if this is a namespace issue where the AutoML dataframe is being interrupted and I need to uniquely name one of the Auto_ml variables below. Thank you. Here is my code and the result:

1) CODE:

2) RESULT:

Activity

Show:
Erin LeDell
October 15, 2020, 6:48 PM
Edited

Sorry, somehow I missed this ticket when it was created! There’s a project_name parameter for AutoML, which will allow you to execute two different AutoML runs on the same dataset on the same H2O cluster. Here, if they are run on the same machine, they will still compete for resources, however.

You can also run two H2O instances on different ports on the same machine. e.g. h2o.init(port = 54321, nthreads = 32) and h2o.init(port = 55555, nthreads = 32)? I think this will use different sets of cores for each H2O instance, but I don’t think it’s guaranteed (the OS will try to balance this). The drawback here is that they can’t share data, so the training set will be duplicated. If you have enough RAM, this is probably better though.

Assignee

Unassigned

Fix versions

None

Reporter

Michael Jules

Support ticket URL

None

Labels

None

Affected Spark version

None

Customer Request Type

None

Task progress

None

ReleaseNotesHidden

None

CustomerVisible

No

Epic Link

Components

Affects versions

Priority

Critical