Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP::StaticFileHandler serve pre-gzipped content #9626

Merged
merged 3 commits into from
Aug 19, 2020

Conversation

waj
Copy link
Member

@waj waj commented Jul 21, 2020

This allows serving .gz files just like other web servers, avoiding on the fly compression, returning the content of foo.txt.gz when foo.txt was requested. This reduces CPU usage and response time. For example, in my machine a 1M javascript file is served in around 35ms (compressed with HTTP::CompressHandler). But the time goes down to around 300us when the .gz file is served instead.

The .gz file is only served if it's newer than the uncompressed file. The modification time from the uncompressed file is still used for cache checks and Etag header.

I found that in macOS, doing gzip -k doesn't keep the exact same modification time, loosing some precision. That's why I allowed up to 1 millisecond in the past for the compressed file and still being served as the static content. I think it's very unlikely this could cause issues in real world.

This PR works great with #9625, now that it allows other handlers to return compressed content.

@asterite
Copy link
Member

Should this be documented in the handler docs? I would never have guessed this happens automatically. What other web servers do this?

@waj
Copy link
Member Author

waj commented Jul 21, 2020

Should this be documented in the handler docs?

Absolutely! Let me add some docs.

What other web servers do this?

NGINX: http://nginx.org/en/docs/http/ngx_http_gzip_static_module.html
Apache: https://httpd.apache.org/docs/2.4/mod/mod_deflate.html#precompressed

WebPack has a module to emit .gz files alongside each asset: https://webpack.js.org/plugins/compression-webpack-plugin/
Rollup also has a plugin to do the same thing: https://www.npmjs.com/package/rollup-plugin-gzip

It's a really common pattern to serve compressed static content.

@waj
Copy link
Member Author

waj commented Jul 21, 2020

@asterite I just added the docs. I also removed the word "simple" because this is probably the most complex handler we have right now in the standard library 😆

Copy link
Member

@asterite asterite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@waj waj added this to the 1.0.0 milestone Jul 21, 2020
Copy link
Member

@straight-shoota straight-shoota left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not entirely sure it's a good idea to stuff more and more stuff into static file handler. It's supposed to be just a really simple tool. IMO such more advanced features should not be provided by stdlib. It just adds more and more complexity and we don't have the resources to maintain a really stable, production-ready webserver implementation.

@waj
Copy link
Member Author

waj commented Jul 24, 2020

I'm not planning to add excessive complexity, but this was really easy to implement and adds real huge benefit. In some cases it might even delay the need for many people to setup more complex deployments with caching layers or separate webservers for static content.

@j8r
Copy link
Contributor

j8r commented Jul 24, 2020

What about brotli and other possible encoding algorithms, too marginally used to be worth supporting?
A simple way to support it is to have a property for the algorithm name and one for the extension.

@waj
Copy link
Member Author

waj commented Jul 30, 2020

@j8r maybe, but I didn't want to add more complexity or change the API at this moment

@bcardiff bcardiff merged commit d5d79dd into crystal-lang:master Aug 19, 2020
@bcardiff
Copy link
Member

FYI this specs fail when executed via docker with the source in a mounted container. The mtime of the file is truncated to seconds upon read and set. This makes the test.txt and test.txt.gz mtime to differ in a whole second 😞 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants