XGBoost gets stuck with 50+ executors (instead of failing outright)

Fixed with XGB 1.2 and fix https://github.com/h2oai/xgboost/commit/7c2686146e489d5a70cc56c0ab3dff745541c334#diff-004c0be4015a26da794093f163d56d8264387c6831562cad1551e513d550117cL52

User resolved issue by upgrading to 3.32.0.3, which had XGBoost upgrade to 1.2 and fix ( )

was not able to reproduce on 3.30.1.x, suggesting upgrade since the code causing the error above was changed in xgboost

Investigation of logs revealed:

there are two bugs: first that the training did not stop because of this error, and second that the error even happened - but that could be an xgboost bug

this is bad because it prevents us from shutting down cleanly