Use dynamic file names for assets

[silverwind]

WIP on dynamic asset filenames which turns /js/index.js into /js/index.5ed90373e37c.js. This is done via webpack-assets-manifest which creates a assets-manifest.json file which is used by the backend to map the filenames.

TODOs:

• Find a solution for development reloads
• Render the resolved filenames back into the template
• Make it work for bindata
• Fix Makefile logic for index.js and index.css
• Gracefully handle absence of assets-manifest.json for integration tests

I'm unsure what the best option is for development rebuilds as the backends need to be able to reload assets-manifest.json when it changes, both inside bindata and in the file system. One simple option could be to only use dynamic asset names during the prod build. Another option could be a SIGUSR1 (SIGHUP is already in use) sent by webpack to the backend, but I don't think it's a portable option.

Fixes: #18522

[wxiaoguang]

Actually the original PR #18348 (/assets/public-{ver}/index.js) was almost done and very simple (+24 −16), and it can handle all cases.

This PR is already much more complex (+190 −28) and still have a lot of TODOs, and it doesn't handle existing JS files without webpack (CodeMirror plugins, PDF.js, etc, the public/vendor/plugins), and it might need hacky methods to handle filenames in templates and during development.

Although I still feel that #18348 is better and more suitable for current Gitea, if this PR can handle all cases correctly without introducing hacky methods, then I would have no objection.

[silverwind]

This PR is already much more complex

More complex, but the proper solution imho

lot of TODOs

Yes, it's WIP

existing JS files without webpack

I plan to remove all files outside webpack in the future, it's just bad practice and CodeMirror can break anytime with the currently outdated plugins.

might need hacky methods to handle filenames in templates and during development

Not hacky imho, and I think I've found a solution for development by Stat()ing the manifest.

[silverwind]

It seems the general recommendation for webpack during development is to not emit hash-based assets (files will pile up in the filesystem until watch is restarted, there is clean option but I found it to be buggy).

Current solution still has it in by checking the mtime of the manifest, but I think I will opt to just follow the recommendation and remove it during development.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 months. Thank you for your contributions.

[silverwind]

Let's keep this open.

[silverwind]

Rebased. Still WIP.

[lunny]

All returns need Unlock

[silverwind]

Development & Production version should enable cache when the asset is the same and disable cache when the asset is not the same. Since cache could be disabled from web browser. It seems it's unnecessary to diff Development & Production versions.

@lunny: time-based cache doesn't work like that. Browser will not even attempt to re-validate a cached resource until cache duration has expired, even if it had already changed on the server. ETag-based cache could do what you want but it would require re-validation (e.g. If-None-Match header) on every request, which will increase server load.

Once cache busting via filename works (this PR), I agree, we can make prod/dev the same, but it's still convenient to not have the dev server cache at all.

[silverwind]

todo: should path.Clean in all cases.

[silverwind]

On topic here: It might be possible make a "light" version of this PR and introduce contenthash for dynamic chunks only, which is really the main cause of the cache issues. Cache busting via query string from HTML works, but the problem happens because dynamic chunks are always served with the same names. I'll try that.

Edit: #20813

[silverwind]

Let's continue this in #20813. I think the solution here with asset manifest is probably too complicated, but will likely be necessary if we ever want to have consistent contenthash in all filenames.