StreamingQueryException: Input byte array has wrong 4-byte ending unit

[trincaog]

When trying to collect Azure Eventhub messages using Spark/Python, I always get the exception "StreamingQueryException: Input byte array has wrong 4-byte ending unit"

Sample code:

conf = {}
conf["eventhubs.connectionString"] = "Endpoint=sb://XXXXXXXXX.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXX=;EntityPath=XXXXXX"
                                      
read_df  = spark.readStream.format("eventhubs").options(**conf).load()
stream = read_df.writeStream.format("console").start()
stream.awaitTermination()

Error:
StreamingQueryException: Input byte array has wrong 4-byte ending unit
=== Streaming Query ===
Identifier: [id = dc06f9ed-73b3-4198-9f68-4f28f78132be, runId = c34f5655-e2d0-43fe-b101-8b58e5b70a14]
Current Committed Offsets: {}
Current Available Offsets: {}

Current State: INITIALIZING
Thread State: RUNNABLE

Versions:
Spark 3.0.1 (Databricks 7.3LTS)
com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.17

[nyaghma]

@trincaog
As it has been mentioned here, for 2.3.15 version and above, the configuration dictionary requires that the connection string be encrypted.

[trincaog]

Thank you!