Memory issue with version v3.667.0

[JeremieDoctrine]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

We have three services using the aws-sdk` v3.667.0
This is the memory usage per version :

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-dynamodb@3.667.0, @aws-sdk/lib-dynamodb@3.667.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v22.4.1

Reproduction Steps

run npm start the container immediately runs out of memory.

Observed Behavior

Memory increases at boot time

Expected Behavior

Memory do not increase at boot time

Possible Solution

No response

Additional Information/Context

No response

[JeremieDoctrine]

I know there is very little information in this ticket. I have no much time to investigate.
But i'm, quite sure this version is bugged. And wanted to create an issue because i'm sure other people will run into the same issue.

[Chrisp1tv]

Hello!
I'm working on a Nodejs (version 22.9.0) project and I'm observing the same issue when using AWS SDK with NodeJS streams since this version, 3.667.0. We can provide more details if it can help to debug the issue :)

[lucavb]

In our case it is also making our Nest app go OOM on startup.

[FelixK-Witt]

I can confirm that our deployments run out of memory during startup too, when upgrading from 3.666.0 to 3.667.0. We're using node/express server.

[MathieuGuillet]

Same here with node JS / Nest and these dependencies:

    "@aws-sdk/client-s3": "^3.667.0",
    "@aws-sdk/credential-providers": "^3.667.0",
    "@aws-sdk/lib-storage": "^3.667.0",
    "@aws-sdk/lib-storage": "^3.667.0",
    "@aws-sdk/rds-signer": "^3.667.0",

[krukid]

same here, the memory leak occurred with these aws deps

aws-sdk@2.1691.0
@aws-sdk/credential-providers@3.667.0

rolling back to @aws-sdk/credential-providers@3.664.0 fixed the issue

[kuhe]

Hi, we are marking 3.666.0 series of @aws-sdk/* packages as latest on NPM while we prepare a fix.

[kuhe]

I'm sorry for not catching this bug. A fix has been made in PR #6555.

We will release the new version later today.

The root cause is calling the memoized credentials provider function in the user agent middleware. The credentials provider function may include an SDK operation to e.g. STS, and during that invocation it loops into the same middleware.

After resolving the immediate issue, we will investigate how to improve test coverage to avoid recurrences.

[kuhe]

https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.668.0 has been released with what I believe is the fix.

That said, does anyone have a more specific reproduction setup for this issue?

[matt-halliday]

https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.668.0 has been released with what I believe is the fix.

That said, does anyone have a more specific reproduction setup for this issue?

I don't have a full repro repo... can probably create one if needed?

We were seeing it in a node API running in K8s with a service role granting GetObject access to a bucket.

A request would come in:

/get-foo/{id}

Our S3Client would go something like this:

try {
    const get = new GetObjectCommand({
        Bucket: 'bucket',
        key: <the-id>,
    });
    // we get here
    const data = await s3Client.send(get);
    return JSON.parse(data.Body);
} catch (err) {
    // we never get here
}

It would crash immediately...

[angusjellis]

We've observed the same behaviour. We have a simple tool that includes the AWS SDK as part of its dependencies.

It instantiates an S3 client before doing anything else, using an OIDC token from GitLab CI.

When the tool runs on a container with 2GB of RAM, if using version3.666 it works as expected, but on version 3.667 we can pinpoint that it runs out of memory as soon as the S3 client is instantiated.

[matt-halliday]

We've observed the same behaviour. We have a simple tool that includes the AWS SDK as part of its dependencies.

It instantiates an S3 client before doing anything else, using an OIDC token from GitLab CI.

When the tool runs on a container with 2GB of RAM, if using version3.666 it works as expected, but on version 3.667 we can pinpoint that it runs out of memory as soon as the S3 client is instantiated.

Interesting, our client instantiated ok, but exploded whenever we tried to use it

Edit: now I remember we were intially getting crashloops on start up, increased resource limits to get around that, and then saw the crashes on calling the client. Fog of war.

[aqeelat]

@kuhe @aws-sdk/types is not updated. Latest is still 3.664.0.

[kuhe]

Not all packages get updated to every version. The clients on version 3.668.0 still use @aws-sdk/types@3.667.0, which is not an affected package. The affected package is @aws-sdk/middleware-user-agent@3.667.0.

[kuhe]

It looks like container credentials may be the precondition, which I'll investigate.

[matt-halliday]

It looks like container credentials may be the precondition, which I'll investigate.

Using a profile, I couldn't recreate locally with a basic benchmark so that's quite possible... I've attached some nasty code if it helps. You can change the client version in package.json, run npm i && npm run benchmark for output in the console.

benchmark.zip

[kuhe]

I believe this is fixed with v3.668.0, so I'll be closing the issue soon unless anyone reports the issue as persisting in v3.668.0, but I'll comment with the root cause details when I'm able to determine the preconditions.

[kuhe]

My testing shows the issue affected assumeRoleWithWebCredentials but not assumeRole. assumeRoleWithWebCredentials usually coincides with usage of process.env.AWS_WEB_IDENTITY_TOKEN_FILE.