Added logging package and fixed issue with API calls not being shown in DEBUG or lower log levels.

[tanmay-db]

Changes

Terraform never had logging configured correctly (used to log all levels irrespective of log level specified). The issue (api calls not being shown in DEBUG log level) surfaced because in the TF version bump to 1.20.9 which had Go SDK version bump to 0.10.0, the PR: databricks/databricks-sdk-go#426 introduced INFO as basic log level leading to debug level never being enabled even if it is passed through the environment variable.

Tests

Unit tests added, Integration tests are running...

Also tested manually

 terraform {
  required_providers {
    databricks = {
      source  = "databricks/databricks"
      version = "1.26.0"
    }
  }
}

provider "databricks" {
    profile = <redacted>
}

data "databricks_catalogs" "all" {}

output "all_catalogs" {
  value = data.databricks_catalogs.all
}

shows the api calls

2023-09-21T03:03:41.340+0200 [DEBUG] provider.terraform-provider-databricks: GET /api/2.1/unity-catalog/catalogs
< HTTP/2.0

• make test run locally
• relevant change in docs/ folder
• covered with integration tests in internal/acceptance
• relevant acceptance tests are passing
• using Go SDK

[tanmay-db]

There seems to be another issue with the provider / how terraform provider is configured. If TF_LOG="" or is not set then INFO level isn't used.

This is true even without these changes.

[tanmay-db]

It works if TF_LOG=INFO is set

[tanmay-db]

I can try to investigate why this is happening but given it was happening before (not sure if it's expected behaviour) and can easily be mitigated with TF_LOG=info being set, we could go ahead with changes and file a bug fix for this to do later.

[mgyucht]

While this would solve the immediate need, I don't think it's the right way to solve this problem. Instead, we should implement the Logger interface from the SDK using the tflog package and set the SDK logger to use that.

[mgyucht]

nit: for this and line 28, we probably should use value receivers rather than pointer receivers. There isn't really a reason to use a pointer receiver, and it categorically prevents nil-pointer errors from ever happening.

[mgyucht]

I don't know that this is great to do in init(), since this assumes that the logging configuration is done fully statically by TF (and I'm a bit nervous to put things in init() that don't have to be there). Maybe we move this to ConfigureContextFunc.

[mgyucht]

Interesting. This probably works fine given the current SDK logging architecture and gets us back to where we were, but I think it's a bit of a hack to try to go around TF logging infra and read from the environment like this. For instance, users can't enable debug logging for just our provider (otherwise, we would also need to check TF_LOG_PROVIDER_DATABRICKS as well).

Another approach would be to define a struct and implement the logging.Logger interface, then configure the SDK to use this logger. This is what is done in the AWS TF provider implementation. They implement a custom logger that implements their Logger interface and then configure the client to use that logger at configuration time. The advantage of this approach is that you let TF determine what gets printed to the log (so, for instance, TF could change the log level while the provider was running, or provide different ways for users to specify log level rather than through environment variables).

[tanmay-db]

Thanks @mgyucht for review, closing this in favour of: #2747. This PR is still WIP, need to clean it up and add tests but it is working fine tested manually.