Revise parallel grid search in respect to early stopping and user cancellation
We need to make sure the less common scenarios of using grid search work well. During work on https://github.com/h2oai/h2o-3/pull/5115 I had to remove attempt to handle user (immediate) user cancellation and early stopping - that wasn't working properly.
We need to handle these cases:
job canceled - I think if someone cancels the job, it should propagate to all the “sub-jobs” (that is not being done)
early stopping - should we finish the models that are still being built - I would probably still finish them
better handling of out-of-time - could it be tied to job (can the job have an automatic stop time? when it appears to be canceled - hence translated to case 1?).