Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce peak memory usage during Lift Export #811

Closed
johnthagen opened this issue Nov 13, 2020 · 1 comment · Fixed by #878
Closed

Reduce peak memory usage during Lift Export #811

johnthagen opened this issue Nov 13, 2020 · 1 comment · Fixed by #878

Comments

@johnthagen
Copy link
Collaborator

johnthagen commented Nov 13, 2020

Currently when the backend exports a Lift file, it:

  • Generates a zip file temporarily on disk (can be many 100s of MB)
  • It reads this into memory
  • It base64 decodes the contents in memory (which expands it ~30%)
  • It sends this to the client in an HTTP response

This leads to peak memory usage of about (export_size x 2.3). It has been observed that for a 600MB export, system memory utilization increased to 2.2GB.

One option is to use NGINX sendfile to allow NGINX to efficiently stream the file back to the client and not load it into the backend memory:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant