Info Logs leak the eventhub connection string

[alex-shchetkov]

Summary: Full eventhub connection string is being printed in the INFO log during Microbatch Execution

• Actual behavior:
  Logger - org.apache.spark.sql.execution.streaming.MicroBatchExecution
  Message - "Using Source [org.apache.spark.sql.eventhubs.EventHubsSource@1e59e6ae] from DataSourceV1 named 'eventhubs' [DataSource(org.apache.spark.sql.SparkSession@b4c1fb2,eventhubs,List(),None,List(),None,Map(eventhubs.consumerGroup -> my-group, eventhubs.connectionString -> Endpoint=sb://my-eventhub.servicebus.windows.net/;SharedAccessKeyName=msl;SharedAccessKey=actual-key-is-here;EntityPath=topic-name, eventhubs.startingPosition -> {"enqueuedTime": "2020-03-30T00:00:00.0000Z", "isInclusive": true}),None)]"

• Expected behavior
  The connection string should not be printed in any logs at any time

• Spark version
  2.4.4

• spark-eventhubs artifactId and version
  com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.14.1

[sjkwak]

@alex-shchetkov - the log message comes from the Spark runtime and I believe the logging was added in Spark runtime 2.4 and it logs the contents of options (for e.g., eventhub config) which is set when initializing a spark session.

[spzSource]

Hi @nyaghma and @sjkwak

I have to raise concern about #491

In the PR connection string are encrypted using library version as a key for AES algorithm. So it's completely insecure, as everybody can decrypt the value. Hence applied fix do not solves initial issue, but just hides it.

Also by forcing user for encrypting connection string, it adds additional troubles when using this library under .NET for Apache Spark (a.k.a .net backend), where it's not possible to directly access to JVM methods.

To be precise, following code doesn't work in .NET backed for Spark:

sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connectionString)

So that it forces to re-implement encrypt logic in .NET (and other spark backends), which is quite odd..

My view is that correct fix would be to apply changes for Spark runtime, so that spark itself should not log sensitive content.