Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure DataBricks, Spark Streaming (Python) doesn't show data on console #324

Closed
rawatsudhir1 opened this issue May 28, 2018 · 2 comments
Closed

Comments

@rawatsudhir1
Copy link

rawatsudhir1 commented May 28, 2018

Environment
DataBricks Runtime :- 4.0 (includes Apache Spark 2.3.0, Scala 2.11)
Python Version :- 2.0
Spark_EventHub Jar :- azure-eventhubs-spark_2.11 3.1

Code

from pyspark.sql.types import *
from pyspark.sql.functions import *
connectionString = "Endpoint=sb://XXXXXX.servicebus.windows.net/;SharedAccessKeyName=iothubowner;SharedAccessKey=XXXXXXXXXXXXXXXXXX;EntityPath=XXXXXXXXXXXXXXX"
ehConf = {}
ehConf['eventhubs.connectionString'] = connectionString
inputStream = spark
.readStream
.format("eventhubs")
.options(**ehConf)
.load()
schema = StructType()
.add("Column1", StringType())
.add("Column2", StringType())
.add("Column3", StringType())
.add("Column4", StringType())
.add("Column5", StringType())
.add("Column6", StringType())
entries = (inputStream
.selectExpr("CAST(body as STRING)")
.select(from_json(col("body"), schema).alias("data")))
entries.writeStream.format("append").format("console").start().awaitTermination()

Issue
When above code executes it's doesn't show data on console but shows in Driver Logs. Here is the screenshot

spark_display_issue

However Scala works fine.

@rawatsudhir1
Copy link
Author

Removing extra comments

@rawatsudhir1
Copy link
Author

closing this as I figured out the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant