New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add `bundle debug terraform` command #1294

Merged

andrewnester merged 29 commits into main from terraform-dependencies

Apr 2, 2024

Contributor

ilia-db commented Mar 18, 2024 •

edited by pietern

Loading

Add bundle debug terraform command. It prints versions of the Terraform and the Databricks Terraform provider. In the text mode it also explains how to setup the CLI in environments with restricted internet access.
Use DATABRICKS_TF_EXEC_PATH env var to point Databricks CLI to the Terraform binary. The CLI only uses it if DATABRICKS_TF_VERSION matches the currently used terraform version.
Use DATABRICKS_TF_CLI_CONFIG_FILE env var to point Terraform CLI config that points to the filesystem mirror for the Databricks provider. The CLI only uses it if DATABRICKS_TF_PROVIDER_VERSION matches the currently used provider version.

Relevant PR on the VSCode extension side: databricks/databricks-vscode#1147

Example output of the databricks bundle debug terraform:

Terraform version: 1.5.5
Terraform URL: https://releases.hashicorp.com/terraform/1.5.5

Databricks Terraform Provider version: 1.38.0
Databricks Terraform Provider URL: https://github.com/databricks/terraform-provider-databricks/releases/tag/v1.38.0

Databricks CLI downloads its Terraform dependencies automatically.

If you run the CLI in an air-gapped environment, you can download the dependencies manually and set these environment variables:

  DATABRICKS_TF_VERSION=1.5.5
  DATABRICKS_TF_EXEC_PATH=/path/to/terraform/binary
  DATABRICKS_TF_PROVIDER_VERSION=1.38.0
  DATABRICKS_TF_CLI_CONFIG_FILE=/path/to/terraform/cli/config.tfrc

Here is an example *.tfrc configuration file:

  disable_checkpoint = true
  provider_installation {
    filesystem_mirror {
      path = "/path/to/a/folder/with/databricks/terraform/provider"
    }
  }

The filesystem mirror path should point to the folder with the Databricks Terraform Provider. The folder should have this structure: /registry.terraform.io/databricks/databricks/terraform-provider-databricks_1.38.0_ARCH.zip

For more information about filesystem mirrors, see the Terraform documentation: https://developer.hashicorp.com/terraform/cli/config/config-file#filesystem_mirror

ilia-db added 6 commits

March 18, 2024 17:28


          Add bundle dependencies command

c3ca75d

- Add `bundle dependencies` command. It prints versions of terraform and
databricks terraform provider.
- Use DATABRICKS_TF_EXEC_PATH env var to point databricks CLI to terraform bin
- Use DATABRICKS_TF_PLUGIN_CACHE_DIR env var to point terraform CLI to the
plugin cache dir


          Check versions and files/dirs

e8ec958


          Update log message

aff7024


          Merge branch 'main' into terraform-dependencies


          Add tests for TF_PLUGIN_CACHE_DIR

d99a003


          Add tests for DATABRICKS_TF_EXEC_PATH

8a3b0b7

ilia-db requested review from pietern and kartikgupta-db

March 19, 2024 10:50

ilia-db marked this pull request as ready for review

March 19, 2024 10:51


          Improve tests

c157a5c

kartikgupta-db reviewed

View reviewed changes

bundle/deploy/terraform/init.go Outdated Show resolved Hide resolved

cmd/bundle/dependencies.go Outdated Show resolved Hide resolved

ilia-db mentioned this pull request

Bundle cli dependencies databricks/databricks-vscode#1147

Merged

ilia-db added 3 commits

March 20, 2024 12:43


          Do not override existing execPath from the bundle

f6142ce


          Improve tests

91924ed


          Close terraform binary file after creation

55c7808

pietern reviewed

View reviewed changes

bundle/deploy/terraform/init.go Outdated Show resolved Hide resolved

bundle/deploy/terraform/init.go Show resolved Hide resolved

bundle/deploy/terraform/init.go Outdated Show resolved Hide resolved

bundle/deploy/terraform/init_test.go Outdated Show resolved Hide resolved

bundle/internal/tf/schema/root.go Show resolved Hide resolved

cmd/bundle/dependencies.go Outdated Show resolved Hide resolved

cmd/bundle/dependencies.go Outdated Show resolved Hide resolved

ilia-db added 8 commits

March 20, 2024 16:39


          Add debug logs, improve tests

1e549a4


          Use TF_CLI_CONFIG_FILE instead of TF_PLUGIN_CACHE_DIR

70334a5

With plugin cache dir terraform still tries to access the registry


          Add plain text output

20ff8e1


          Remove unnecessary test

58bbd71


          Formatting

c5df7af


          Better template message

51175c9


          Merge branch 'main' into terraform-dependencies

7e42269


          Update json names

92e5cef

ilia-db requested a review from pietern

March 22, 2024 13:32

ilia-db changed the title ~~Add bundle dependencies command~~ Add bundle internal-dependencies command

juliacrawf-db reviewed

View reviewed changes

cmd/bundle/dependencies.go Outdated Show resolved Hide resolved

juliacrawf-db reviewed

View reviewed changes

cmd/bundle/dependencies.go Outdated Show resolved Hide resolved

juliacrawf-db reviewed

View reviewed changes

cmd/bundle/dependencies.go Outdated Show resolved Hide resolved

juliacrawf-db reviewed

View reviewed changes

cmd/bundle/dependencies.go Outdated Show resolved Hide resolved

ilia-db added 2 commits

March 28, 2024 14:47


          Merge branch 'main' into terraform-dependencies

8884a41


          Update tests

d99642f

shreyas-goenka reviewed

View reviewed changes

bundle/deploy/terraform/init.go Outdated Show resolved Hide resolved

bundle/deploy/terraform/init.go

+              	if err != nil {
+              		if os.IsNotExist(err) {
+              			log.Debugf(ctx, "%s at %s does not exist, ignoring %s", envVarName, envValue, versionVarName)
+              			return "", nil

Contributor

shreyas-goenka Mar 28, 2024

How does this tie in with Databricks Terminals? If these are only set in those then we don't have to bother with long lifetimes of env vars in terminals.

bundle/deploy/terraform/init.go Outdated Show resolved Hide resolved

bundle/deploy/terraform/init.go

+              	if err != nil {
+              		if os.IsNotExist(err) {
+              			log.Debugf(ctx, "%s at %s does not exist, ignoring %s", envVarName, envValue, versionVarName)
+              			return "", nil

Contributor

shreyas-goenka Mar 28, 2024 •

edited

Loading

we can't change them after the initial setup

For my knowledge, updating the extensions requires reloading the IDE, and then reloading the IDE should refresh the env vars right?

cmd/bundle/debug/terraform.go

+              		case flags.OutputText:
+              			cmdio.Render(cmd.Context(), dependencies.Terraform)
+              		case flags.OutputJSON:
+              			buf, err := json.MarshalIndent(dependencies, "", "  ")

Contributor

shreyas-goenka Mar 28, 2024

Nice! We can also use this in the air-gapped docker image for the CLI to query for TF versions required.

ilia-db and others added 2 commits

April 2, 2024 10:23


          Update bundle/deploy/terraform/init.go

Co-authored-by: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com>


          Move env vars into consts

138a9e1

ilia-db requested review from shreyas-goenka, pietern and juliacrawf-db

April 2, 2024 09:13

pietern reviewed

View reviewed changes

bundle/deploy/terraform/pkg.go Outdated Show resolved Hide resolved

Contributor

pietern commented Apr 2, 2024

Please update the PR summary to be up to date with the most recent revision.

Everything else LGTM.

ilia-db added 2 commits

April 2, 2024 14:07


          Fix typo

d921a82


          Merge branch 'main' into terraform-dependencies

d157a2c

ilia-db requested a review from pietern

April 2, 2024 12:30

pietern approved these changes

View reviewed changes

pietern requested review from andrewnester and juliacrawf-db and removed request for juliacrawf-db

April 2, 2024 12:43

ilia-db added this pull request to the merge queue

andrewnester removed this pull request from the merge queue due to a manual request

andrewnester approved these changes

View reviewed changes

andrewnester added this pull request to the merge queue

Merged via the queue into main with commit 079c416

5 checks passed

andrewnester deleted the terraform-dependencies branch

April 2, 2024 13:03

andrewnester added a commit that referenced this pull request


          Release v0.217.0

1f85841

CLI:
 * Added `auth describe` command ([#1244](#1244)).
 * Fixed message for successful auth describe run ([#1336](#1336)).

Bundles:
 * Make bundle validation print text output by default ([#1335](#1335)).
 * Use UserName field to identify if service principal is used ([#1310](#1310)).
 * Allow unknown properties in the config file for template initialization ([#1315](#1315)).
 * Remove support for DATABRICKS_BUNDLE_INCLUDES ([#1317](#1317)).
 * Make `bundle.deployment` optional in the bundle schema ([#1321](#1321)).
 * Add allow list for resources when bundle `run_as` is set ([#1233](#1233)).
 * Fix the generated DABs JSON schema ([#1322](#1322)).
 * Make bundle loaders return diagnostics ([#1319](#1319)).
 * Add `bundle debug terraform` command ([#1294](#1294)).
 * Allow specifying CLI version constraints required to run the bundle ([#1320](#1320)).

Internal:
 * Retain location information of variable reference ([#1333](#1333)).
 * Define `dyn.Mapping` to represent maps ([#1301](#1301)).
 * Return `diag.Diagnostics` from mutators ([#1305](#1305)).
 * Fix flaky test in `libs/process` ([#1314](#1314)).
 * Move path field to bundle type ([#1316](#1316)).
 * Load bundle configuration from mutator ([#1318](#1318)).
 * Return diagnostics from `config.Load` ([#1324](#1324)).
 * Return warning for nil primitive types during normalization ([#1329](#1329)).
 * Include `dyn.Path` in normalization warnings and errors ([#1332](#1332)).
 * Make normalization return warnings instead of errors ([#1334](#1334)).

API Changes:
 * Added `databricks lakeview migrate` command.
 * Added `databricks lakeview unpublish` command.
 * Changed `databricks ip-access-lists get` command . New request type is .

OpenAPI commit e316cc3d78d087522a74650e26586088da9ac8cb (2024-04-03)
Dependency updates:
 * Bump github.com/databricks/databricks-sdk-go from 0.36.0 to 0.37.0 ([#1326](#1326)).

andrewnester mentioned this pull request

Release v0.217.0 #1337

Merged

github-merge-queue bot pushed a commit that referenced this pull request


          Release v0.217.0 (#1337)

6ac45e8

Breaking Change:
* Add allow list for resources when bundle `run_as` is set
([#1233](#1233)).
* Make bundle validation print text output by default
([#1335](#1335)).

CLI:
* Added `auth describe` command
([#1244](#1244)).
* Fixed message for successful auth describe run
([#1336](#1336)).

Bundles:
* Use UserName field to identify if service principal is used
([#1310](#1310)).
* Allow unknown properties in the config file for template
initialization ([#1315](#1315)).
* Remove support for DATABRICKS_BUNDLE_INCLUDES
([#1317](#1317)).
* Make `bundle.deployment` optional in the bundle schema
([#1321](#1321)).
* Fix the generated DABs JSON schema
([#1322](#1322)).
* Make bundle loaders return diagnostics
([#1319](#1319)).
* Add `bundle debug terraform` command
([#1294](#1294)).
* Allow specifying CLI version constraints required to run the bundle
([#1320](#1320)).

Internal:
* Retain location information of variable reference
([#1333](#1333)).
* Define `dyn.Mapping` to represent maps
([#1301](#1301)).
* Return `diag.Diagnostics` from mutators
([#1305](#1305)).
* Fix flaky test in `libs/process`
([#1314](#1314)).
* Move path field to bundle type
([#1316](#1316)).
* Load bundle configuration from mutator
([#1318](#1318)).
* Return diagnostics from `config.Load`
([#1324](#1324)).
* Return warning for nil primitive types during normalization
([#1329](#1329)).
* Include `dyn.Path` in normalization warnings and errors
([#1332](#1332)).
* Make normalization return warnings instead of errors
([#1334](#1334)).
API Changes:
 * Added `databricks lakeview migrate` command.
 * Added `databricks lakeview unpublish` command.
* Changed `databricks ip-access-lists get` command . New request type is
.

OpenAPI commit e316cc3d78d087522a74650e26586088da9ac8cb (2024-04-03)
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.36.0 to 0.37.0
([#1326](#1326)).

ilia-db added a commit to databricks/databricks-vscode that referenced this pull request


          Bundle cli dependencies (#1147)

5b4f896

Bundle terraform binaries with the extension and expose them with env
vars to the CLI and the terminal

Depends on databricks/cli#1294

shreyas-goenka mentioned this pull request

Add docker images for the CLI #1353

Merged

github-merge-queue bot pushed a commit that referenced this pull request


          Add docker images for the CLI (#1353)

5140a9a

## Changes
This PR makes changes to support creating a docker image for the CLI
with the `terraform` dependencies built in. This is useful for customers
that operate in a network-restricted environment. Normally DABs makes
API calls to registry.terraform.io to setup the terraform dependencies,
with this setup the CLI/DABs will rely on the provider binaries bundled
in the docker image.

### Specifically this PR makes the following changes:
----------------
Modifies the CLI release workflow to publish the docker images in the
Github Container Registry. URL:
https://github.com/databricks/cli/pkgs/container/cli.

We use docker support in `goreleaser` to build and publish the images.
Using goreleaser ensures the CLI packaged in the docker image is the
same release artifact as the normal releases. For more information see:
1. https://goreleaser.com/cookbooks/multi-platform-docker-images
2. https://goreleaser.com/customization/docker/

Other choices made include:
1. Using `alpine` as the base image. The reason is `alpine` is a small
and lightweight linux distribution (~5MB) and an industry standard.
2. Not using [docker
manifest](https://docs.docker.com/reference/cli/docker/manifest) to
create a multi-arch build. This is because the functionality is still
experimental.

------------------

Make the `DATABRICKS_TF_VERSION` and `DATABRICKS_TF_PROVIDER_VERSION`
environment variables optional for using the terraform file mirror.
While it's not strictly necessary to make the docker image work, it's
the "right" behaviour and reduces complexity. The rationale is:
- These environment variables here are needed so the Databricks CLI does
not accidentally use the file mirror bundled with VSCode if it's
incompatible. This does not require the env vars to be mandatory.
context: #1294
- This makes the `Dockerfile` and `setup.sh` simpler. We don't need an
[entrypoint.sh script to set the version environment
variables](https://medium.com/@leonardo5621_66451/learn-how-to-use-entrypoint-scripts-in-docker-images-fede010f172d).
This also makes using an interactive terminal with `docker run -it ...`
work out of the box.

## Tests

Tested manually.

--------------------

To test the release pipeline I triggered a couple of dummy releases and
verified that the images are built successfully and uploaded to Github.
1. https://github.com/databricks/cli/pkgs/container/cli
3. workflow for release:
https://github.com/databricks/cli/actions/runs/8646106333

--------------------

I tested the docker container itself by setting up
[Charles](https://www.charlesproxy.com/) as an HTTP proxy and verifying
that no HTTP requests are made to `registry.terraform.io`

Before:
FYI, The Charles web proxy is hosted at localhost:8888.
```
shreyas.goenka@THW32HFW6T bundle-playground % rm -r .databricks
shreyas.goenka@THW32HFW6T bundle-playground % HTTP_PROXY="http://localhost:8888" HTTPS_PROXY="http://localhost:8888" cli bundle deploy
Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
```
<img width="1275" alt="Screenshot 2024-04-11 at 3 21 45 PM"
src="https://github.com/databricks/cli/assets/88374338/15f37324-afbd-47c0-a40e-330ab232656b">

After:
This time bundle deploy is run from inside the docker container. We use
`host.docker.internal` to map to localhost on the host machine, and -v
to mount the host file system as a volume.
```
shreyas.goenka@THW32HFW6T bundle-playground % docker run -v ~/projects/bundle-playground:/bundle -v ~/.databrickscfg:/root/.databrickscfg -it --entrypoint /bin/sh -e HTTP_PROXY="http://host.docker.internal:8888" -e HTTPS_PROXY="http://host.docker.internal:8888" --network host ghcr.io/databricks/cli:latest-arm64
/ # cd /bundle/
/bundle # rm -r .databricks/
/bundle # databricks bundle deploy
Uploading bundle files to /Users/shreyas.goenka@databricks.com/.bundle/bundle-playground/default/files...
Deploying resources...
Updating deployment state...
Deployment complete!
```

<img width="1275" alt="Screenshot 2024-04-11 at 3 22 54 PM"
src="https://github.com/databricks/cli/assets/88374338/2a8f097e-734b-4b3e-8075-c02e98a1b275">

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

kartikgupta-db kartikgupta-db left review comments

pietern pietern approved these changes

andrewnester andrewnester approved these changes

shreyas-goenka Awaiting requested review from shreyas-goenka

juliacrawf-db Awaiting requested review from juliacrawf-db

Labels

None yet