Memory leak going from 3.2.2 to 3.3.0

[smeyfroi]

Our app uses sidekiq ent (latest version) with the redis-rb driver. It also access redis directly using redis-rb to do stuff using redis as a datastore.

After updating gems last week, I saw the active memory on the rails server increase quickly and eventually kill the box.

See attached: the section before the big dip (redeploy) is what happens with redis 3.3.0, where the green [active memory] line eventually maxes out and crashes the server. The section after the redeploy is what happens with redis 3.2.2. The 3.2.2 'active memory' line tends to level out over time roughly at the level where I've taken this screenshot.

Here's another screenshot that shows the different a bit more clearly. Here, the sections up to a redeploy early on 22/4 are 'normal' with active memory sitting reasonably well around 1.17GB. Then after the update to redis-rb you can see the green 'active memory' line climbs and climbs quickly, until catastrophic failure when the kernel kills the process and the cycle continues.

I think this is due to the work that sidekiq does in polling work queues in the background, because once I'd disabled most of our sidekiq workers from doing any work by doing return from their perform methods, the memory-chomping continued. So I think this effect may be visible with sidekiq just polling in the background etc. We use sidekiq-ent's unique jobs functionality which also polls redis in the background.

I'm not sure how to help debug this but want to raise it because it's probably an important issue unless I'm doing something really weird in my app (which is entirely possible).

Downgrading to 3.2.2 does definitely fix the issue with no other changes (I'm watching the graphs now: they look completely normal again.)

Are you aware of any changes that may have introduced a problem like this? Can I help you debug it?

[badboy]

Thanks for reporting this. From the top of the head I don't know what could cause this, though we were investigating a (not yet fixed) leak in redis-rb. The change was rather big from 3.2.2 to 3.3.0. We're investigating that.

[smeyfroi]

Cool. Let me know if there's anything I can do to help. It's easy enough for me to redeploy this test server and see the issue if you need it. :-)

[kamen-hursev]

I confirm we experience the same issue after upgrading to 3.3.0. We are using Sidekiq, as well, and though it is not heavily loaded its memory usage climbs up more than what we usually monitor.

In a couple of hours I can confirm if the issue really goes away after I just rolled back to 3.2.2.

[djanowski]

I'm not a Sidekiq user, can you gist a runnable self-contained example so
we can look into it?
On Apr 26, 2016 8:35 AM, "Kamen Hursev" notifications@github.com wrote:

I confirm we experience the same issue after upgrading to 3.3.0. We are
using Sidekiq, as well, and though it is not heavily loaded its memory
usage climbs up more than what we usually monitor.

In a couple of hours I can confirm if the issue really goes away after I
just rolled back to 3.2.2.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#612 (comment)

[badboy]

@kamen-hursev, @smeyfroi: Are you using redis-rb in combination with hiredis-rb or without?

[kamen-hursev]

I'm not using hiredis-rb. Just redis-rb.

[badboy]

that confirms my suspicion. I ran a very tiny script through memory_profile: gist.
Seems like we're allocating quite some Timeout instances.
A run with hiredis shows no bigger memory usage.

If possible, can either of you run with hiredis? gem install hiredis, then pass :driver => :hiredis to Redis.new.

[kamen-hursev]

I won't be able to test hiredis in the next 2 weeks. We observe the problem on production only (the test environments are even less loaded) and now is not a suitable time to make experiments on production. So I can report back after minimum two weeks.

[badboy]

No problem then. I don't expect you to test in production. :)
I hope we have this fixed within less than 2 weeks though

[smeyfroi]

Hi, I can test that out on my test server. Might be the morning (UK time) time though.

[badboy]

@smeyfroi Thanks!

[smeyfroi]

OK I just redeployed with redis3.3.0 and hiredis0.6.1. The graphs look good so far but I'll let it run for a while and report back.

Is hiredis recommended underneath redis-rb for performance? (I never understood this part of the ecosystem.)

[badboy]

I don't have benchmarks at hand, but yeah, performance is the one reason you might want to use hiredis.
We now know the reason of the leak in the ruby connector in v3.3.0 though and will release a fix ASAP.

[smeyfroi]

Great! Well done tracking that down.

[smeyfroi]

Quick note to say that my test server running redis3.3.0 and hiredis0.6.1 looks completely normal at the end of the day here. Guess that helps confirm your forthcoming fix!

[fertobar]

Hello, any update about this issue?
I'm using 3.2.2. do you recommend me to upgrade to other version or will better to wait?

[badboy]

@fertobar If you use hiredis-rb you should be fine.
I'm gonna pull in the required changes for the pure-ruby connector this week.

This is lying around for far too long already :/

[maia]

@badboy Is there any news regarding the memory leak of 3.3.x? Thanks!
(I'm running the pure ruby connector on heroku)

[badboy]

I finally merged the fix now, though can't release the gem on rubygems right now. I'll push it out once I get access.

[YesThatAllen]

So, this is fixed in Release v3.3.1 ?

[badboy]

Yes, it should be.

[maia]

Thanks a lot, much appreciated!

[badboy]

If someone can confirm this, it would be appreciated. If anything else comes up please report back, until then I'm gonna close this ticket.

[mperham]

Looks better, thanks!

[soylent]

Version 3.3.1 resolves the issue, thanks!

v 3.3.0:

allocated memory by gem
-----------------------------------
  26268727  2.3.1/lib
     70216  redis-3.3.0
      6920  [snip]
      4516  other

v 3.3.1:

allocated memory by gem
-----------------------------------
     67559  redis-3.3.1
      6920  [snip]
      6127  2.3.1/lib
      4516  other