Esoteric Bug with regards to authentication via requests headers= argument being overwritten by .netrc

[ewengillies]

Describe the bug

It's an odd one that set me back around a day: I hope to save people a similar amount of frustration.

When running a python model on an all purpose cluster, I would get this error:

10:54:28    Error getting status of cluster.
10:54:28     b'{"error_code": "403", "message": "Invalid access token."}'

despite putting a brand new token into the profiles.yml for the target corresponding all purpose cluster.

Turns out that the my .netrc file was to blame. It had an expired token for the same host:. Also turns out that the requests library overrides authentication methods passed through header= argument with the contents of the .netrc file, as stated here: https://requests.readthedocs.io/en/latest/user/authentication/#netrc-authentication

To be 100% clear, my .netrc file looked like this (replacing sensitive info below):

.netrc:

machine <my-workspace>.cloud.databricks.com
login token
password <expired_token>

My .profiles contained a target like:

all_purpose:
  type: databricks
  host: <my-workspace>.cloud.databricks.com
  http_path: <sql_path_for_all_purpose_cluster>
  catalog: <catalog>
  schema: <schema>
  token: <valid_token>
  threads: 1

Then the weird part:

• requests.py does pass the <valid_token> in to the headers argument in the get_cluster_status call.
• With that said, requests.py would give me an error from this method because it defaulted to my .netrc token: https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/python_submissions.py#L245

Steps To Reproduce

Fairly easy to trigger:

• Put an expired or invalid token into the .netrc file.
  • Note that using a .netrc file is encouraged in the Databricks API documentation
• Put a valid token in the ~/.dbt/profiles.yml for an all purpose cluster that we want to use to run a python model.
• Run the model, get the error from here: https://github.com/databricks/dbt-databricks/blob/main/dbt/adapters/databricks/python_submissions.py#L255

Expected behavior

I'd expect the token provided in my profiles.yml to take precedence.

Screenshots and log output

Not needed IMO, already know the cause.

System information

Relevant versions are:

• dbt-databricks==1.5.0
• requests==2.28.1

Additional context

I would say the solution is to ensure the dbt-databricks code uses the auth argument instead. It seems a "bearer token" auth method doesn't come included in requests (which seems odd but okay). To this end there is a good SO solution to this: https://stackoverflow.com/a/58055668

I would also highlight that while this feels like it might only be a me problem, Databricks has lots of docs around using .netrc file, as linked before: Databricks API documentation. Doesn't seem too far-fetched someone else will fall into this trap, and its a hard one to get out of.

[andrefurlan-db]

Thanks for the bug report, we clearly missed this edge case. We'll address it soon. For the time being we'll not honour the .netrc file in dbt-databricks. Especially now with OAuth support, saving tokens locally should not be as important.

[ewengillies]

Thanks for your response. I think it makes sense to ignore the .netrc file, its strange how right now its implicitly included thanks to requests.

Will look into OAuth support as well, thanks!

[susodapop]

I've written the fix and opened #338

[benc-db]

Closing as fixed. If the issue still exists, please reopen.

[lennartkats-db]

@benc-db This issue still appears to happen as the fix wasn't merged yet, could we reopen? This issue breaks materialized views when a .netrc file is present. See #338 (comment).

[lennartkats-db]

Btw here's a link to the bug in requests that seems to cause this issue: psf/requests#3929