Extension sigv4auth signature failure

[arpitjindal97]

Component(s)

extension/sigv4auth

What happened?

Description

I have set credentials in ~/.aws/credentials with credential_process. OTEL is able to read them but sigv4 signing is failing.

Steps to Reproduce

otelconfig.yaml

receivers:
  otlp:
    protocols:
      http:
exporters:
  logging:
    loglevel: debug
extensions:
  http_forwarder:
    ingress:
      endpoint: 0.0.0.0:7070
    egress:
      endpoint: https://vpc-es-opensearch-deployment-1-xxxx.eu-central-1.es.amazonaws.com
      auth:
        authenticator: sigv4auth
  sigv4auth:
    region: eu-central-1
    service: es
service:
  extensions: [http_forwarder, sigv4auth]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: []
      exporters: [logging]

~/.aws/credentials

[default]
credential_process = bash ~/.aws/print_cred.sh

print_cred.sh outputs:

{
  "expiration_time": "2023-11-26T22:52:24.270279066Z",
  "lease_id": "aws-poc/sts/arn_aws_iam__1111_role_es_opensearch_deployment_1_eu_central_1_Kibana/xxxx",
  "ttl": 3599,
  "AccessKeyId": "xxxx",
  "SecretAccessKey": "xxx",
  "SessionToken": "xxxx",
  "Version": 1,
  "Expiration": "2023-11-26T21:16:12+00:00"
}

Expected Result

OTEL should start and any http request to port 7070 should be forwarded to opensearch domain with correct sigv4 signature.

Actual Result

{
  "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'GET\n/_dashboards\n\naccept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8\naccept-encoding:gzip, deflate, br\naccept-language:en-GB,en-IN;q=0.9,en-NL;q=0.8,en-US;q=0.7,en;q=0.6\ncache-control:max-age=0\nconnection:close\nhost:vpc-es-opensearch-deployment-1-bxg6u34rjbakn2v2dmuc452yhq.eu-central-1.es.amazonaws.com\nkbn-xsrf:reporting\nosd-xsrf:reporting\nsec-ch-ua:\"Brave\";v=\"119\", \"Chromium\";v=\"119\", \"Not?A_Brand\";v=\"24\"\nsec-ch-ua-mobile:?0\nsec-ch-ua-platform:\"macOS\"\nsec-fetch-dest:document\nsec-fetch-mode:navigate\nsec-fetch-site:none\nsec-fetch-user:?1\nsec-gpc:1\nupgrade-insecure-requests:1\nvia:HTTP/1.1 localhost:7070\nx-amz-date:20231126T223720Z\nx-amz-security-token:FwoGZXIvYXdzEFAaDEN6jK4PtDHvP9z/fSLkAau/XteUk99Ul6/acM9EQViW7csOd9XVUlk3X/zbRVSBN2JPp5tOVM27CWbYuZu36fEFM1vtYT1VkMgKxx5njwceOf/ZrpVvdJCaoBiIkKQu4ZuT/nMcfsNZr75xTqU8BQaTYDUGYCT5x9Ocq1MqGDJo7mMxLXY5zW5TscN3QNte9UPOUgFuJlmcL0epe5IpG6nCxgjvOGcPTX7uc7io6+AnWTohectKdUcDc1OLEuegyJGrqVDpZ4+ysZnb4hbUf8dEY4pw3gNhBuuWXrcdtVveht1rKlLFfzPJSNin6Z25Eg9wIyjHhY+rBjIt3gp98644tcBEWJI7cQa3vzU9ys6lqHbE7obSkWulUkQ3J9lEco8GvZk/ixP6\n\naccept;accept-encoding;accept-language;cache-control;connection;host;kbn-xsrf;osd-xsrf;sec-ch-ua;sec-ch-ua-mobile;sec-ch-ua-platform;sec-fetch-dest;sec-fetch-mode;sec-fetch-site;sec-fetch-user;sec-gpc;upgrade-insecure-requests;via;x-amz-date;x-amz-security-token\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20231126T223720Z\n20231126/eu-central-1/es/aws4_request\n412727f18ea5a7cb182ae45e9262cf33a6ea7c424a2acf81b8a57893b90732d6'\n"
}

Collector version

v0.89.0

Environment information

No response

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• extension/sigv4auth: @Aneurysm9 @erichsueh3

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bryan-aguilar]

Hi @arpitjindal97,

How are you executing the collector? Are you using using an otel collector image? If so, by default the collector is built with a scratch base image and does not contain a shell. This means it would have not be able to execute the bash script to generate the credentials.

[arpitjindal97]

I'm compiling the otel binary from source and then executing it. It is able to find the credentials correctly.

Another wired behaviour i noticed is If i call curl localhost:7070 from terminal then it's not an issue but from browser it's an issue. I have also tried different browser. I suspect some header aren't playing well with SigV4

[arpitjindal97]

Also, if i add this header in config:

http_forwarder:
    ingress:
      endpoint: 0.0.0.0:7070
    egress:
      endpoint: https://vpc-es-opensearch-deployment-1-xxxx.eu-central-1.es.amazonaws.com
      headers:
        connection: close
      auth:
        authenticator: sigv4auth

Then for some requests browser works but fails on few

[bryan-aguilar]

1. Can you provide more details on your environment?
2. Can you provide a reproducible example?
3. Have you tried this with static credentials rather than credential_process in your ~/.aws/credentials file. This would help eliminate whether it's related to the credential process auth chain.

[arpitjindal97]

I'm on MacBook

1. Clone the repo
2. execute make otelcontribcol
3. You will end up creating a binary file
4. Put your credentials in ~/.aws/credentials in any format
5. Create a config.yaml as mentioned initially in this thread.
6. Have a working OpenSearch Domain
7. Try to visit localhost:7070/_dashboards from browser

[bryan-aguilar]

Put your credentials in ~/.aws/credentials in any format

Does this mean that it also fails when using a static set of credentials rather than credential process? Because if so that may mean an issue with the components in use and not sigv4auth.

[arpitjindal97]

Yes, I think the issue might not be with sigv4auth but rather http_forwarder. Can you help in debugging this component?

[github-actions]

Pinging code owners for extension/httpforwarder: @atoulme @rmfitzpatrick. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• extension/sigv4auth: @Aneurysm9 @erichsueh3
• extension/httpforwarder: @atoulme @rmfitzpatrick

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.