Respect SparkSession of a current environment

Description

None

Activity

Show:
Neema Mashayekhi
February 20, 2020, 9:43 PM

Customer found in Azure Databricks that when two notebooks connected to the same DataBricks cluster, TempView cannot be created in one of the notebooks. Issue is resolved when they restart the cluster.

Testing was done at H2O. The first notebook to connect to H2O is able to access the TempView tables (including ones created by second notebook).

Neema Mashayekhi
February 20, 2020, 9:45 PM

Related issue:

Neema Mashayekhi
February 20, 2020, 9:47 PM

Produced an error in Python and Scala:

Create one notebook, run:

No error occurs

 

Create a second notebook and connect to same cluster, Repeat the script above changing TempView name:

Error:

AnalysisException: 'Table or view note found: output_df_2; line 1 pos 38

It will error saying TempView not found. This is because the first notebook has the H2O cluster pointing to its SQL context and all TempView tables are being saved to the first notebooks SQL context.

Recommendation: Separate the caching of data for the two notebooks (although they connect to the same H2O cluster)

Assignee

Jakub Hava

Reporter

Jakub Hava

Labels

None

CustomerVisible

No

testcase 1

None

testcase 2

None

testcase 3

None

h2ostream link

None

Affected Spark version

None

AffectedContact

None

AffectedCustomers

None

AffectedPilots

None

AffectedOpenSource

None

Support Assessment

None

Customer Request Type

None

Support ticket URL

End date

None

Baseline start date

None

Baseline end date

None

Task progress

None

Task mode

None

ReleaseNotesHidden

None

Fix versions

Priority

Major
Configure