plugin crashes destroying BQ dataset panic: Error reading level state: strconv.ParseInt: parsing "1691737052926": value out of range

[intotecho]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.5.5
on windows_386

• provider registry.terraform.io/hashicorp/google v4.77.0
• provider registry.terraform.io/hashicorp/google-beta v4.77.0
• provider registry.terraform.io/hashicorp/random v3.5.1
• provider registry.terraform.io/hashicorp/time v0.9.1

Affected Resource(s)

Terraform Configuration Files

This is not the complete config, just a submodule.

module "bigquery" {
  source  = "terraform-google-modules/bigquery/google"
  version = "~> 6.1"

  dataset_id                  = "xxxx"
  dataset_name                = "xxxx"
  description                 = "Dataset"
  project_id                  = local.project_id
  location                    = local.region
  default_table_expiration_ms = 3600000
  max_time_travel_hours       = 168
  dataset_labels = {
    env = local.environment
  }
}

Debug Output

Panic Output

module.tracemark_project.module.bigquery.google_bigquery_dataset.main: Refreshing state... [id=projects/ul-gs-u-903249-09-prj/datasets/tracemark]
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by the current configuration.
│
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically suggests to use it as part of an error message.
╵
╷
│ Error: Plugin did not respond
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-google_v4.77.0_x5.exe plugin:

panic: Error reading level state: strconv.ParseInt: parsing "1691737052926": value out of range

goroutine 210 [running]:
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*ResourceData).get(0x1774b9a0, {0x178009a0, 0x1, 0x1}, 0x8)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/resource_data.go:553 +0x279
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*ResourceData).State(0x1774b9a0)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/resource_data.go:351 +0x3a7
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ShimInstanceStateFromValue(0x16883b90, {{{0x3dba9a0, 0x1776ffa0}}, {0x31e6fe0, 0x1675a600}})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/resource.go:599 +0x1d9
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0x16c496a0, {0x3dba2c0, 0x171a7218}, 0x165c1a40)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/grpc_provider.go:587 +0x23f
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ReadResource({0x16c079e0, 0x16c07a20, {0x16cd8260, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
        github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_ReadResource.go:26 +0x151
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0x1701a3c0, {0x3dba2c0, 0x171a6e58}, 0x16d185d0)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:748 +0x4c5
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x36ceda0, 0x1701a3c0}, {0x3dba2c0, 0x171a6e58}, 0x17755440, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:349 +0x18a
google.golang.org/grpc.(*Server).processUnaryRPC(0x17010d80, {0x3dbecfc, 0x16f74240}, 0x165200a0, 0x16dfc978, 0x54e6b48, 0x0)
        google.golang.org/grpc@v1.56.2/server.go:1337 +0xed7

Expected Behavior

Destroy the dataset.

Actual Behavior

Panic

Steps to Reproduce

1.terraform destroy, or terraform destroy -target=module.tracemark_project.module.bigquery.google_bigquery_dataset.main

Important Factoids

I have a complex module to configure a project. It includes creating a bigquery dataset.
I refactored it and also renamed the parent module, leading to everything under the old name being destroyed. I have target destroyed all the resources in the state, so only this resource remains and some data resources.

After identifying it was the dataset, I deleted the dataset from the BQ console. But this did not change the panic output.

References

• #0000

[intotecho]

Since the resource has been deleted, I removed it from the state with
tf state rm module.tracemark_project.module.bigquery.google_bigquery_dataset.main Removed module.tracemark_project.module.bigquery.google_bigquery_dataset.main Successfully removed 1 resource instance(s).

Now tf destroy works without crashing.

[edwardmedia]

@intotecho is it possible to repro? Is it running on Windows?

[zli82016]

Hello, @intotecho , I cannot reproduce the issue with the following configuration in the provider 4.77.0.

resource "google_bigquery_dataset" "dataset" {
  dataset_id                  = "example_dataset"
  friendly_name               = "test"
  description                 = "This is a test description"
  location                    = "US"
  default_table_expiration_ms = 3600000
  max_time_travel_hours       = 168
    labels = {
    env = "default"
  }
}

Can you please reproduce the issue and provide more logs? I cannot figure out what filed caused the issue without more information. Thanks.

[intotecho]

After removing the resource from the state, and deleting it from the console, I was able to rebuild it.
I can also target destroy it and recreate it the resource so I am unable to reproduce it now.
I wonder if it was something like the state got corrupted when the authentication expired during an apply.

[intotecho]

On reflection, the refreshed API Key will need a new name, since they are not deleted in GCP but hang around as deleted keys for 30 days preventing terraform from creating a new key with the same name.
So the solution may be to add a random value to the key display name.

[intotecho]

The problem resurfaced in another terraform project, I think I found the culprit.

I upgraded terraform from 1.4.0 to 1.5.5 and the google provider from 4.60 to 4.7.7 but in the process I switched terraform from windows_amd64 to windows_x86.

terraform init migrate-state 
terraform init -upgrade
terraform plan

The crash was similar, and also around a bigquery dataset.

I checked the version of terraform, and also my previous version

terraform --version
Terraform v1.5.5
on windows_386
+ provider registry.terraform.io/hashicorp/google v4.77.0
+ provider registry.terraform.io/hashicorp/google-beta v4.77.0

terraform_1_4.exe --version
Terraform v1.4.5
on windows_amd64
+ provider registry.terraform.io/hashicorp/google v4.77.0
+ provider registry.terraform.io/hashicorp/google-beta v4.77.0

echo $PROCESSOR_ARCHITECTURE
AMD64

I switched to the windows_amd64 version and it resolved the crash issue.

terraform plan
...
module.operations_project.google_bigquery_dataset.dataset: Refreshing state... [id=...]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Plugin did not respond
│
│   with module.operations_project.google_bigquery_dataset.dataset,
│   on ..\..\05_-_operations\main.tf line 75, in resource "google_bigquery_dataset" "dataset":
│   75: resource "google_bigquery_dataset" "dataset" {
│
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
│ details.
│ Error: googleapi: Error 403: The caller does not have permission, forbidden
│ 
│   with module.operations_project.google_monitoring_monitored_project.projects_monitored["project_id"],
│   on ..\..\05_-_operations\main.tf line 119, in resource "google_monitoring_monitored_project" "projects_monitored":
│  119: resource "google_monitoring_monitored_project" "projects_monitored" {
│
│ Error: Plugin did not respond
│
│   with module.operations_project.google_logging_folder_sink.environment-sink,
│   on ..\..\05_-_operations\main.tf line 136, in resource "google_logging_folder_sink" "environment-sink":
│  136: resource "google_logging_folder_sink" "environment-sink" {
│
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
│ details.
╵

Stack trace from the terraform-provider-google_v4.77.0_x5.exe plugin:

panic: Error reading level state: strconv.ParseInt: parsing "1671430042825": value out of range

goroutine 257 [running]:
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*ResourceData).get(0x1791c5f0, {0x17929138, 0x1, 0x1}, 0x8)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/resource_data.go:553 +0x279
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*ResourceData).State(0x1791c5f0)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/resource_data.go:351 +0x3a7
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ShimInstanceStateFromValue(0x16a223f0, {{{0x3b8a9a0, 0x179284a0}}, {0x2fb6fe0, 0x16a61940}})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/resource.go:599 +0x1d9
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0x16828be0, {0x3b8a2c0, 0x17180780}, 0x16a60c80)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.24.0/helper/schema/grpc_provider.go:587 +0x23f
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ReadResource({0x16810000, 0x16810040, {0x16d7aa10, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
        github.com/hashicorp/terraform-plugin-mux@v0.8.0/tf5muxserver/mux_server_ReadResource.go:26 +0x151
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0x16f9e2d0, {0x3b8a2c0, 0x171803c0}, 0x16df4b10)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/tf5server/server.go:748 +0x4c5
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x349eda0, 0x16f9e2d0}, {0x3b8a2c0, 0x171803c0}, 0x17920240, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.14.3/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:349 +0x18a
google.golang.org/grpc.(*Server).processUnaryRPC(0x1701c480, {0x3b8ecfc, 0x165a4120}, 0x16534640, 0x16f37200, 0x52b6b48, 0x0)
        google.golang.org/grpc@v1.56.2/server.go:1337 +0xed7
google.golang.org/grpc.(*Server).handleStream(0x1701c480, {0x3b8ecfc, 0x165a4120}, 0x16534640, 0x0)
        google.golang.org/grpc@v1.56.2/server.go:1714 +0xa17
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        google.golang.org/grpc@v1.56.2/server.go:959 +0xb1
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.56.2/server.go:957 +0x178

Error: The terraform-provider-google_v4.77.0_x5.exe plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[slevenick]

Looks like the architecture is the issue here. windows_amd64 is 64-bit, windows_386 which causes the problem is 32-bit, which will not be able to handle the given value as an integer. I'm seeing some older issues where this popped up in other fields:

#6518
#13775

I think the root cause is here: hashicorp/terraform-plugin-sdk#469 which I will reopen due to this still occurring on modern versions of the SDK.

[slevenick]

I'm going to close this out in favor of #6518, but I'll add a link to this to show it still happens

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.