Reduce size of bundle / release

[garygreen]

I've attempted to download the Windows binary as found at:

https://artifacts.elastic.co/downloads/kibana/kibana-6.7.0-windows-x86_64.zip

This leads to a 187MB zipped file. I attempted to unzip it and after 10 mins of it still running I gave up and moved onto a more "minimal" install version:

https://artifacts.elastic.co/downloads/kibana/kibana-oss-6.7.0-windows-x86_64.zip

This is now a 77MB file and has taken around 5 minutes to unzip and during that process I noticed it's re-created virutally every module in the npm registry in node_modules.

So long story short:

• Is there any way the Kibana team can reduce the size of the bundle and improve the experience in getting up and running with Kibana quicker?
• Is it really necessary to include everything under node_modules in a release?

[garygreen]

As a point of interest, here's what the tree looks like after unzipping. Most of the space is taken up by node_modules. If you take a look inside, it's got babel and other dependencies which I would assume is only applicable during packaging/bundling?

[kobelb]

@garygreen at this point in time, all of the node_modules which we include in the .zip are required to run the Kibana server and enable other ancillary operations. There are quite a few initiatives underway to reduce the bundle sizes, and remove the need to ship certain node_modules in production versions, but they're all rather complex and intricate changes that we're making incrementally.

[elasticmachine]

Pinging @elastic/kibana-operations

[mistic]

@garygreen thanks for raising that problem. Actually this a thing we are aware of and also already have plans to solve in the upcoming future!

[garygreen]

Ok thanks for the quick response, it's much appreciated. I searched for any other issues on it and didn't see anything mentioning the build size / user experience in installation, so thought I'd open something. Would be great if you can leave this open so can keep track of progress.

At the moment, I'm not too comfortable installing Kibana on our production server given the amount of time it takes and size of the bundle, but would like to revisit once it's been optimized. Thanks once again 👍

[mistic]

Yes we can let it opened! As soon we have progresses on it, I'll post here!

[tylersmalley]

Unzipping on Windows is also pretty horrendous. I would recommend using 7zip.

[garygreen]

Unzipping on Windows is also pretty horrendous. I would recommend using 7zip.

It takes around 10 mins+ on my machine (Intel i5 2500k, not a bad CPU by any means) to unzip the full package, even when using 7zip.

I wonder if the compression level is a factor in how long it takes to unzip? I tried to find in Kibana src where it defines a compression level for zip files but couldn't. At some point when you reach a certain level it doesn't really decrease the zip size but massively increases decompression time, so wonder if that is playing a role.

In any case, I think 95% of the libraries in node_modules aren't required in the release.

[spalger]

@garygreen I think it's a little unfair to say that 95% of the libraries in node_modules aren't required.

I suspect it's the disk I/O, and potentially windows defender or other security scans, that cause the decompression time to scale based on the number of files in the archive.

We currently strip all the files we think are unnecessary from node_modules here: https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/clean_tasks.js#L64-L171 and yarn is doing its best to flatten the dependency tree as much as possible.

@jbudz also has #32760 which will flatten even more of the different module versions, that work is ongoing.

@mistic also has POC implementations of a static file system that would dramatically reduce the number of files in the repo by shipping them as a single file, of which little parts can be loaded when a specific module is requested.

[garygreen]

@spalger apologies, that did sound a little dramatic. It was a comment purely based on a visual glance of the output from WinDirStat and looking at the overall size of the top modules.

I did a basic calculation and worked out that at least 50MB is taken up with "build" dependencies, babel, webpack, less, etc. That's approx 50%.

At the moment it's a pretty poor first user experience when decompressing, it shouldn't really take 10mins+ to install, but it could just be a Windows issue like you say. Though just FYI - I don't have anything special going on my machine, I don't use a virus scanner, just bog standard windows settings and I'm unzipping to a SSD.

Glad to see that work is being made to reduce the bundle size though, PRs look promising 👍

[spalger]

Thanks, I understand.

worked out that at least 50MB is taken up with "build" dependencies, babel, webpack, less, etc.

Getting rid of these modules is another project: removing the optimizer from the productions builds (#7322). This is currently necessary for plugin installation, which basically causes a re-build of the front-end assets on startup so that the new code can use the existing modules and integrate into the system. We're working on removing it.

[huan086]

I'm looking at the deb package for 7.9. Many node_modules, e.g. @elastic/eui, comes with lib, es and dist. If the Kibana source consistently reference just one of them, the other 2 could be removed.

Another big reduction can be done similarly for monaco-editor, which has dev, esm and min. dev is not used, Kibana references both esm and min currently, and ideally only esm or min is used.

[tylersmalley]

There has been a lot of progress on this since this issue was originally opened. Since then, we have removed the optimizer (Webpack) from production which removed any dependencies needed to build. This resulted in us going from 137k files, down to 57k. I understand there is still lots of room for improvement, but we will use issues to track those individual efforts going forward.