Unable to run multiple H2O AutoML processes simultaneously
Description
As a machine learning engineer, I need to be able to run multiple H2o AutoML processes simultaneously. I get the below error when attempting to run multiple instances on the same H2o server – but this should be possible. I'm wondering if this is a namespace issue where the AutoML dataframe is being interrupted and I need to uniquely name one of the Auto_ml variables below. Thank you. Here is my code and the result:
1) CODE:
2) RESULT:
Activity
Sorry, somehow I missed this ticket when it was created! There’s a project_name parameter for AutoML, which will allow you to execute two different AutoML runs on the same dataset on the same H2O cluster. Here, if they are run on the same machine, they will still compete for resources, however.
You can also run two H2O instances on different ports on the same machine. e.g. h2o.init(port = 54321, nthreads = 32) and h2o.init(port = 55555, nthreads = 32)? I think this will use different sets of cores for each H2O instance, but I don’t think it’s guaranteed (the OS will try to balance this). The drawback here is that they can’t share data, so the training set will be duplicated. If you have enough RAM, this is probably better though.
Assignee
Fix versions
Reporter
Support ticket URL
Labels
Affected Spark version
Customer Request Type
Task progress
ReleaseNotesHidden
CustomerVisible
Epic Link
Components
Affects versions
Priority
