adding the patch into H2O 3 code to re-connect to the h2o nodes

Description

customer was facing the issue of cluster disconnecting and 401 authentication error
work around was using the script below and customer would like to add as patch fix for coming up h2o-3 version:

def _query_job_status_safe(self):
import time
from h2o.exceptions import H2OConnectionError, H2OResponseError, H2OServerError
result = None
attempts = 0
last_err = None
while attempts < 300:
try:
attempts += 1
result = h2o.api("GET /3/Jobs/%s" % self.job_key)
self.job_poll_success = True # only retry if there was at least one OK response
break
except (H2OConnectionError, H2OResponseError, H2OServerError) as e:
last_err = e
if self.job_poll_success:
print("Job request failed %s, will retry after 3s." % e.args[0])
time.sleep(3)
else:
raise e
if result:
return result
else:
raise last_errdef _refresh_job_status(self):
if self._poll_count <= 0: raise StopIteration("")
jobs = self._query_job_status_safe()
self.job = jobs["jobs"][0] if "jobs" in jobs else jobs["job"][0]
self.status = self.job["status"]
self.progress = self.job["progress"]
self.exception = self.job["exception"]
self.warnings = self.job["warnings"] if "warnings" in self.job else None
self._poll_count -= 1

  1. Sometimes the server may report the job at 100% but still having status "RUNNING" – we work around this

  2. by showing progress at 99% instead. Sometimes the server may report the job at 0% but having status "DONE",

  3. in this case we set the progress to 100% manually.
    from h2o.utils.shared_utils import clamp
    if self.status == "CREATED": self.progress = 0
    if self.status == "RUNNING": self.progress = clamp(self.progress, 0, 0.99)
    if self.status == "DONE": self.progress = 1
    if self.status == "FAILED": raise StopIteration("failed")
    if self.status == "CANCELLED": raise StopIteration("cancelled by the server")
    return self.progressfrom h2o.job import H2OJob
    H2OJob._refresh_job_status = _refresh_job_status
    H2OJob._query_job_status_safe = _query_job_status_safe

suports:
https://support.h2o.ai/a/tickets/98708
https://support.h2o.ai/a/tickets/98776

Fixed

Assignee

Michal Kurka

Fix versions

Reporter

Retima Dangol

Support ticket URL

Labels

None

Affected Spark version

None

Customer Request Type

None

Task progress

None

ReleaseNotesHidden

None

CustomerVisible

No

Priority

Major