Use scratch webpack config #9536

cwillisf · 2024-03-14T22:53:44Z

Proposed Changes

Use our new scratch-webpack-configuration to standardize the way scratch-gui builds and fix the kinds of problems that are blocking #9533 and other updates.

Reason for Changes

General dependency updates, especially in scratch-svg-renderer, and especially especially things related to jsdom, mean we need to change some things about the way we webpack. Some of it's related to webpack 5, but not all of it. I suspect we'll soon need to move to ESM for everything... this set of changes can probably be seen as a precursor to that.

I took the opportunity to make further changes. The main one is setting it up so that webpack can build "deeply" -- that is, when webpack builds (for example) scratch-gui, it'll access the source code of scratch-vm and scratch-svg-renderer (and more in the future) instead of building their bundled forms. This should give us a more optimal bundle overall. In particular, it will hopefully prevent problems where we're embedding multiple versions of the same Scratch library (like scratch-svg-renderer) into scratch-gui or scratch-www. That kind of issue was a major component of the "gray question mark issue" from a while back.

Some of the reasoning for the particulars of the webpack configuration is available here: https://github.com/scratchfoundation/scratch-webpack-configuration

There's some further cleanup to do, such as moving some of these settings into the shared configuration, but I think this is a good (and working!) time to pause for feedback.

Test Coverage

Affects code already under test. Does not increase or decrease test coverage.

Browser Coverage

Check the OS/browser combinations tested (At least 2)

Coming soon

Mac

Chrome
Firefox
Safari

Windows

Chrome
Firefox
Edge

Chromebook

Chrome

iPad

Safari

Android Tablet

Chrome

BREAKING CHANGE: changes layout of output files and browser compatibility

cwillisf · 2024-03-14T22:55:09Z

.babelrc

-        ["@babel/preset-env", {"targets": {"browsers": ["last 3 versions", "Safari >= 8", "iOS >= 8"]}}],
+        "@babel/preset-env",


Moved to .browserslistrc

cwillisf · 2024-03-14T22:56:16Z

package.json

    "build": "npm run clean && webpack",
-    "clean": "rimraf ./build && mkdirp build && rimraf ./dist && mkdirp dist",
+    "clean": "rimraf ./build ./dist",


webpack makes its own directories, so we don't need to use mkdirp to do that.

cwillisf · 2024-03-14T22:59:27Z

package.json

-      "\\.(jpg|jpeg|png|gif|eot|otf|webp|svg|ttf|woff|woff2|mp4|webm|wav|mp3|m4a|aac|oga)$": "<rootDir>/test/__mocks__/fileMock.js",
+      "\\.(jpg|jpeg|png|gif|eot|otf|webp|svg|ttf|woff|woff2|mp4|webm|wav|mp3|m4a|aac|oga)\\??$": "<rootDir>/test/__mocks__/fileMock.js",


We were already telling jest to mock imports for these file types. This tells jest to keep mocking them even if they include an optional question mark at the end (\??, but with an extra layer of escaping).

cwillisf · 2024-03-15T15:15:36Z

src/lib/backpack/sound-payload.js

-import soundThumbnail from '!base64-loader!./sound-thumbnail.jpg';
+import soundThumbnail from '!base64-loader!./sound-thumbnail.jpg?';


This makes me a bit sad. What I really wanted was a way to say, "hey webpack, please use the new fancy Asset Modules feature with all assets EXCEPT those handled by some other loader." That way, things like import foo from '!base64-loader!./foo.png' could use base64-loader and things like import foo from './foo.png' would use Asset Modules. I tried a few ways to hook that up and couldn't get it working, so instead I went with this: imports with certain file extensions (including jpg and png, etc.) are handled as Asset Modules only if they explicitly say so (with a query like ?url) or have no query at all (not even ?, an empty query).

cwillisf · 2024-03-15T15:19:04Z

src/lib/define-dynamic-block.js

-import ArgumentType from 'scratch-vm/src/extension-support/argument-type';
-import BlockType from 'scratch-vm/src/extension-support/block-type';
+import {ArgumentType, BlockType} from 'scratch-vm';


Reaching into a module to grab some arbitrary script is no longer allowed, so I adjusted scratch-vm to export these through the main entry point. In the future, I might make these named, non-default exports in scratch-vm's index.js (once that's ESM) and/or make them separately-specified entry points in the exports section of scratch-vm's package.json.

cwillisf · 2024-03-15T15:19:27Z

test/unit/util/themes.test.js

-            global.XMLSerializer = () => ({
-                serializeToString
-            });
+            class XMLSerializer {
+                constructor () {
+                    this.serializeToString = serializeToString;
+                }
+            }


Fixes an "is not a constructor" error

cwillisf · 2024-03-15T15:20:00Z

webpack.config.js

+                Buffer: require.resolve('buffer/'),
+                stream: require.resolve('stream-browserify')


These are no longer automatically polyfilled in webpack 5.

cwillisf · 2024-03-15T15:26:09Z

webpack.config.js

-        optimization: {
-            splitChunks: {
-                chunks: 'all',
-                name: 'lib.min'
+            path: path.resolve(__dirname, 'dist')
+        }
+    });
+
+// build the examples and debugging tools in `build/`
+const buildConfig = baseConfig.clone()
+    .merge({
+        devServer: {
+            client: {
+                progress: true
            },
-            runtimeChunk: {
-                name: 'lib.min'
-            }


The previous version expressed more control over the way webpack would chunk shared code, but letting webpack figure it out leads to more optimal results. Also, using a single output (lib.min) as the chunk for the webpack runtime and the splitChunks target and a named entry point (old line 130) is no longer allowed by webpack 5. The new configuration is simpler and, IMO, better.

cwillisf · 2024-03-15T15:26:41Z

webpack.config.js

+    }));
+
+// Skip building `dist/` unless explicitly requested
+// It roughly doubles build time and isn't needed for `scratch-gui` development


On my main dev computer, building each target takes a little over 30 seconds, or about 1 minute if you do both (saves like 2-3 seconds).

cwillisf · 2024-03-15T15:57:04Z

webpack.config.js

@@ -1,64 +1,56 @@
-const defaultsDeep = require('lodash.defaultsdeep');


This file is slightly easier to view if you set "Hide whitespace" (or add &w=1 to the URL)

…-scratch-webpack-config Use scratch webpack config

renovate bot and others added 13 commits March 11, 2024 08:08

chore(deps): update dependency webpack to v5

a4d8125

chore(deps): update dependency webpack-cli to v5

96a53e8

chore(deps): update dependency webpack-dev-server to v5

da8d81c

fix(deps): update dependency scratch-vm to v4

bfb634d

chore(deps): update dependency babel-loader to v9

cf9fa06

chore(deps): update dependency html-webpack-plugin to v5

9ebf34f

Merge branch 'develop' into use-scratch-webpack-config

d963343

build!: use scratch-webpack-configuration

f1438f7

BREAKING CHANGE: changes layout of output files and browser compatibility

test: fix XMLSerializer mock

f7ca856

test: fix file mocks for empty query strings

b315db9

fix(deps): update scratch-{svg-renderer,vm,webpack-configuration}

48fb28a

fix(build): bring back BUILD_MODE now that the dust is settling

3b89677

fix: remove unnecessary babel-jest transform config

df34896

cwillisf requested review from colbygk and AlexisGoodfellow March 14, 2024 22:53

cwillisf commented Mar 14, 2024

View reviewed changes

cwillisf commented Mar 15, 2024

View reviewed changes

cwillisf added 2 commits March 18, 2024 13:39

build: move more settings into shared webpack config

b18652d

fix(build): add exports field to package.json

43a1e8f

cwillisf merged commit bbdeb00 into develop Mar 29, 2024
7 checks passed

cwillisf deleted the use-scratch-webpack-config branch March 29, 2024 20:10

github-actions bot pushed a commit that referenced this pull request Mar 29, 2024

Build for bbdeb00 Merge pull request #9536 from scratchfoundation/use…

86ec5cd

…-scratch-webpack-config Use scratch webpack config

This was referenced May 21, 2024

Port number in readme is wrong #9598

Closed

Run on localhost:8602 ScratchAddons/ScratchAddons#7456

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use scratch webpack config #9536

Use scratch webpack config #9536

cwillisf commented Mar 14, 2024

cwillisf Mar 14, 2024

cwillisf Mar 14, 2024

cwillisf Mar 14, 2024

cwillisf Mar 15, 2024

cwillisf Mar 15, 2024

cwillisf Mar 15, 2024

cwillisf Mar 15, 2024

cwillisf Mar 15, 2024

cwillisf Mar 15, 2024 •

edited

Loading

cwillisf Mar 15, 2024

		["@babel/preset-env", {"targets": {"browsers": ["last 3 versions", "Safari >= 8", "iOS >= 8"]}}],
		"@babel/preset-env",

		"\\.(jpg\|jpeg\|png\|gif\|eot\|otf\|webp\|svg\|ttf\|woff\|woff2\|mp4\|webm\|wav\|mp3\|m4a\|aac\|oga)$": "<rootDir>/test/__mocks__/fileMock.js",
		"\\.(jpg\|jpeg\|png\|gif\|eot\|otf\|webp\|svg\|ttf\|woff\|woff2\|mp4\|webm\|wav\|mp3\|m4a\|aac\|oga)\\??$": "<rootDir>/test/__mocks__/fileMock.js",

		import soundThumbnail from '!base64-loader!./sound-thumbnail.jpg';
		import soundThumbnail from '!base64-loader!./sound-thumbnail.jpg?';

		Buffer: require.resolve('buffer/'),
		stream: require.resolve('stream-browserify')

		@@ -1,64 +1,56 @@
		const defaultsDeep = require('lodash.defaultsdeep');

Use scratch webpack config #9536

Use scratch webpack config #9536

Conversation

cwillisf commented Mar 14, 2024

Proposed Changes

Reason for Changes

Test Coverage

Browser Coverage

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwillisf Mar 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwillisf Mar 15, 2024 •

edited

Loading