This document provides an overview of how this project previously updated the Google Terraform Provider (TPG). It serves as a resource for individuals interested in understanding past update procedures and gaining insights into potential future strategies.
We are actively evaluating the optimal long-term strategy for managing Terraform as a dependency. In the interim, for those requiring modifications to the Terraform provider code, please refer to the README.ChangingTerraform.md document for detailed instructions.
- Find the latest release TF provider version tag from the
TF provider’s GitHub release page.
This will be of the format
v5.X.Y.
Read through the provider’s release notes and make note of any aspects that may cause us difficulty with upgrading to this version (especially breaking changes). - Run
git subtree pull --prefix third_party/github.com/hashicorp/terraform-provider-google-beta https://github.com/hashicorp/terraform-provider-google-beta.git v5.X.Y --squash
from the repo’s root. This will update the provider code to the specified version. - Git may prompt you on merge CONFLICT, carefully review and resolve the CONFLICT to complete the git merge.
- Run
make vet
to check if the updated code compiles. - Run
make generate manifests resource-docs generate-go-client
to update all the generated files. - Run
python3 scripts/generate-field-diffs-tf-upgrade/generate_field_changes.py
to generate diff summary of our CRDs. Check "Setting up generate_field_changes tool" section below to set up if this is your first time using it. (Double check withgit diff config/crds/resources
and look through any changes). There may be special situations (as described below) that fields added by the TF provider need to be configured manually. If so, modify the resources’ service mappings accordingly and rerun the command from step 3.- Fields may be implicit resource references that we need to KRM-ify:
- Modify the resources’ service mappings to configure the references.
- (Optional) Consider adding the reference fields in testdata and samples if it is an important ask from users or belongs to a core user case.
- Fields may not exist in the underlying GCP resource, and we need
to turn them into directives: Add the fields into the
directives
list in the service mapping.directives
fields do not exist in resource CRD. - Fields may not exist in the underlying GCP resource, but are returned
by Terraform on read, we need to add them into the
mutableButUnreadableFields
list in the service mapping. The fields are unreadable but the spec value of the fields can be modified. - Fields can't be supported or will result in the suboptimal UX (e.g.
multi-kind reference in the legacy style) due to lack of feature
support, and we need to ignore the fields right now: Add the fields into
the
ignoredFields
list in the service mapping, file a GitHub issue to support this field, and add a TODO with the issue ID right above the entry in the ignoredFields list.
- Fields may be implicit resource references that we need to KRM-ify:
- Rerun
python3 scripts/generate-field-diffs-tf-upgrade/generate_field_changes.py
and save the output change summary so it can be used in step 11. - Run
make test
to run the unit tests.- IAM policy support may have been added to existing resources. If so, add IAM support to the resource’s service mapping. If there is an issue with IAM support and it can’t be added, make a tracking GitHub issue and block it temporarily in the service mapping test code.
- If any new directive fields are added, you might run into issues with
ResourceSkeleton’s
TestNewFromAsset
andTestNewFromURI
tests. Any lingering fields in the expected vs. actual resource skeleton will indicate which fields are suspect. If it makes sense for the field to become a directive, add it to the resource’s corresponding servicemapping. - Golden files may need to be updated. If so, run the relevant test with
the
WRITE_GOLDEN_OUTPUT
env-var to regenerate the golden file. The two tests in particular that may require this are:WRITE_GOLDEN_OUTPUT=1 go test -v ./pkg/crd/fielddesc -test.run TestOutputMatches
WRITE_GOLDEN_OUTPUT=1 go test -v ./pkg/crd/template -test.run TestSpecAndStatusToYAML
- Run
make ready-pr
to ensure the PR is ready to be sent out. - Run
git commit -m "Update TF provider to [version]"
(replacing[version]
with the desired TF version. - Run
git commit --amend
and paste the change summary in the commit message. - Push the commits to the topic branch of your forked repo. Send a PR for review.
- There are chances that presubmit tests in GitHub Action might fail due to the
TF provider bugs in the latest version. Perform a judgment call on the next
step following the below recommendations.
- If you understand the bug well and want to unblock the TF provider upgrade quickly to delivery some user requested features, follow the instructions here to create a TF provider patch to fix the bug locally. Surface the issue separately to the terraform team by creating an issue in terraform-provider-google repository if this bug has not been reported.
- If this bug requires Terraform domain knowledge to fix, report the issue in the terraform-provider-google repo as instructed above, and punt the TF provider upgrade. You should bring this topic as a new issue for further discussion.
On your dev machine, run the following commands:
sudo apt-get install pip
pip3 install gitpython
pip3 install deepdiff