We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Environment DataBricks Runtime :- 4.0 (includes Apache Spark 2.3.0, Scala 2.11) Python Version :- 2.0 Spark_EventHub Jar :- azure-eventhubs-spark_2.11 3.1
Code
from pyspark.sql.types import * from pyspark.sql.functions import * connectionString = "Endpoint=sb://XXXXXX.servicebus.windows.net/;SharedAccessKeyName=iothubowner;SharedAccessKey=XXXXXXXXXXXXXXXXXX;EntityPath=XXXXXXXXXXXXXXX" ehConf = {} ehConf['eventhubs.connectionString'] = connectionString inputStream = spark .readStream .format("eventhubs") .options(**ehConf) .load() schema = StructType() .add("Column1", StringType()) .add("Column2", StringType()) .add("Column3", StringType()) .add("Column4", StringType()) .add("Column5", StringType()) .add("Column6", StringType()) entries = (inputStream .selectExpr("CAST(body as STRING)") .select(from_json(col("body"), schema).alias("data"))) entries.writeStream.format("append").format("console").start().awaitTermination()
Issue When above code executes it's doesn't show data on console but shows in Driver Logs. Here is the screenshot
However Scala works fine.
The text was updated successfully, but these errors were encountered:
Removing extra comments
Sorry, something went wrong.
closing this as I figured out the issue.
No branches or pull requests
Environment
DataBricks Runtime :- 4.0 (includes Apache Spark 2.3.0, Scala 2.11)
Python Version :- 2.0
Spark_EventHub Jar :- azure-eventhubs-spark_2.11 3.1
Code
from pyspark.sql.types import *
from pyspark.sql.functions import *
connectionString = "Endpoint=sb://XXXXXX.servicebus.windows.net/;SharedAccessKeyName=iothubowner;SharedAccessKey=XXXXXXXXXXXXXXXXXX;EntityPath=XXXXXXXXXXXXXXX"
ehConf = {}
ehConf['eventhubs.connectionString'] = connectionString
inputStream = spark
.readStream
.format("eventhubs")
.options(**ehConf)
.load()
schema = StructType()
.add("Column1", StringType())
.add("Column2", StringType())
.add("Column3", StringType())
.add("Column4", StringType())
.add("Column5", StringType())
.add("Column6", StringType())
entries = (inputStream
.selectExpr("CAST(body as STRING)")
.select(from_json(col("body"), schema).alias("data")))
entries.writeStream.format("append").format("console").start().awaitTermination()
Issue
When above code executes it's doesn't show data on console but shows in Driver Logs. Here is the screenshot
However Scala works fine.
The text was updated successfully, but these errors were encountered: