[BUG] S3 bucket metrics are AWS 'region' dependent "NaN"

[ryanpxyz]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Here is a snippet of data that is presented to our Prometheus for scraping:

aws_s3_bucket_size_bytes_sum{account_id="123456789012",dimension_BucketName="ic-xxxx",dimension_StorageType="StandardStorage",name="arn:aws:s3:::ic-xxxx",region="eu-central-1"} NaN
aws_s3_bucket_size_bytes_sum{account_id="123456789012",dimension_BucketName="id-aaaa",dimension_StorageType="StandardStorage",name="arn:aws:s3:::id-aaaa",region="eu-west-1"} 8.58689413e+08
aws_s3_bucket_size_bytes_sum{account_id="123456789012",dimension_BucketName="id-bbbb",dimension_StorageType="StandardStorage",name="arn:aws:s3:::id-bbbb",region="eu-central-1"} NaN
aws_s3_bucket_size_bytes_sum{account_id="123456789012",dimension_BucketName="id-cccc",dimension_StorageType="StandardStorage",name="arn:aws:s3:::id-cccc",region="us-east-1"} 7140`

• We see currently that eu-central-1 resident buckets are presented with "NaN" values.
• We see identical behaviour for other S3 metrics e.g. aws_s3_number_of_objects_maximum.
• We see consistent and correct values & content for all aws_s3_info bucket metrics.

Expected Behavior

All discovered buckets in all regions should present real numbers for scraping by Prometheus.

Steps To Reproduce

• I am working with the following configuration:

    - type: s3
      regions:
        - eu-central-1
        - eu-west-1
        - eu-west-2
        - us-east-1
        - us-east-2
      addCloudwatchTimestamp: false
      period: 86400
      length: 259200
      metrics:
        - name: NumberOfObjects
          statistics:
            - Maximum
          additionalDimensions:
            - name: StorageType
              value: AllStorageTypes
        - name: BucketSizeBytes
          statistics:
            - Sum
          additionalDimensions:
            - name: StorageType
              value: StandardStorage
        - name: BytesUploaded
          statistics:
            - Average
          period: 60
          length: 300
          additionalDimensions:
            - name: FilterId
              value: EntireBucket

• I have reproduced this behaviour with the following YACE versions:
  • "v0.42.0-alpha"
  • "v0.38.0-alpha"
  • "v0.33.0-alpha"
• I have added debug to logging and can confirm that at least in the logs real number data is extracted from AWS CloudWatch - here is a snippet:

{\n      Id: \"id_7316112639016562625\",\n      Label: \"ic-xxxx AllStorageTypes NumberOfObjects\",\n      StatusCode: \"PartialData\",\n      Timestamps: [2022-11-13 00:00:00 +0000 UTC],\n      Values: [299]\n    }
{\n      Id: \"id_2276924059018283294\",\n      Label: \"ic-xxxx StandardStorage BucketSizeBytes\",\n      StatusCode: \"PartialData\",\n      Timestamps: [2022-11-13 00:00:00 +0000 UTC],\n      Values: [3.075721e+06]\n    }
{\n      Id: \"id_7316112639016562625\",\n      Label: \"ic-xxxx AllStorageTypes NumberOfObjects\",\n      StatusCode: \"Complete\"\n    }
{\n      Id: \"id_2276924059018283294\",\n      Label: \"ic-xxxx StandardStorage BucketSizeBytes\",\n      StatusCode: \"Complete\"\n    }


{\n      Id: \"id_7316112639016562625\",\n      Label: \"ic-xxxx AllStorageTypes NumberOfObjects\",\n      StatusCode: \"PartialData\"\n    }
{\n      Id: \"id_2276924059018283294\",\n      Label: \"ic-xxxx StandardStorage BucketSizeBytes\",\n      StatusCode: \"PartialData\"\n    }
{\n      Id: \"id_7316112639016562625\",\n      Label: \"ic-xxxx AllStorageTypes NumberOfObjects\",\n      StatusCode: \"PartialData\"\n    }
{\n      Id: \"id_2276924059018283294\",\n      Label: \"ic-xxxx StandardStorage BucketSizeBytes\",\n      StatusCode: \"PartialData\"\n    }
{\n      Id: \"id_7316112639016562625\",\n      Label: \"ic-xxxx AllStorageTypes NumberOfObjects\",\n      StatusCode: \"PartialData\",\n      Timestamps: [2022-11-14 00:00:00 +0000 UTC],\n      Values: [299]\n    }
{\n      Id: \"id_2276924059018283294\",\n      Label: \"ic-xxxx StandardStorage BucketSizeBytes\",\n      StatusCode: \"PartialData\",\n      Timestamps: [2022-11-14 00:00:00 +0000 UTC],\n      Values: [3.075721e+06]\n    }

Anything else?

• I have tried adding, removing & re-adding tags to the affected buckets.
• I have verified the IAM role as being correct.
• I have played with the AWS CloudWatch statistics (sum, maximum etc) and this didn't produce any change.

[ryanpxyz]

Any ideas, updates? Or is this 'cancelled due to lack of interest'? :-)

[oladuwek]

I have also faced this bug. Is there any workaround or information on how to fix it?

config:

apiVersion: v1alpha1
discovery:
  jobs:
    - type: AWS/S3
      regions: [us-west-1]
      searchTags:
        - key: Name
          value: core-archive-prod
      period: 60
      length: 60
      metrics:
        - name: AllRequests
          statistics: [Sum]

root@ip:~# curl -s localhost:9201/metrics | grep s3
# HELP aws_s3_all_requests_sum Help is not implemented yet.
# TYPE aws_s3_all_requests_sum gauge
aws_s3_all_requests_sum{account_id="123456789",dimension_BucketName="core-archive-prod",dimension_FilterId="test-status",name="arn:aws:s3:::core-archive-prod",region="us-west-1"} NaN
# HELP aws_s3_info Help is not implemented yet.
# TYPE aws_s3_info gauge
aws_s3_info{name="arn:aws:s3:::core-archive-prod",tag_Name="core-archive-prod"} 0

[morremeyer]

Can you check again with v0.61.0 or newer? We faced the same issue, which has been resolved with v0.61.0.