Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes 2 memory leak on search indexing.
The memory profile have been sampled with the following configuration:
Tooling
The profiling script is: https://gist.github.com/noirbizarre/7ea0b0526814131752e6902a4254df78
It requires:
memory-profiler
objgraph
matplotlib
graphviz
Run it with:
Render graph with:
mprof plot output.dat -t "Title" -o output.png
Memory profile from current master
Max memory usage: ~4G
udata
SlugField
The udata
SlugField
instances were keeping a reference to their parent document usingself.instance
. This reference was stored in a class instance which is instanciated once on document definition and so the garbage collector was never able to collect document instances using a slug field (as there was always a reference on it)This has been fixed by refactoring the
SlugField
population into a signal handler and not keeping a static feference on the instance.Memory profile with the
SlugField
fixMax memory usage: ~3.6G
MongoEngine querysets and
no_cache()
MongoEngine default behavior is to cache any queryset just to avoid round trip on "count then query".
This behavior cause a massive memory leak while iterating over a queryset generator.
The fix is to add a
no_cache()
call before the loop.This fix might be beneficial on other code parts.
Memory profile with
no_cache()
Max memory usage: ~300M
MongoEngine 0.17.0
It doesn't fix the memory leak but the update was applied just to verify that it has no impact (See this PR).
As you can see, there is some meory drop and the hole reindexing take much more longer.
The update has been removed from the PR until the cause of these drops is found.
Memory profile
Max memory usage: ~300M