add basic webpack loader support #3284

sokra · 2023-01-12T11:49:01Z

add the minimum to run a simple raw-loader (see test case)

add experimental.turbopackWebpackLoaders to next.config.js
- key is extension like ".mdx", value is an array of loaders like ["mdx-loader"]

vercel · 2023-01-12T11:49:10Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
turbo-site	✅ Ready (Inspect)	Visit Preview	💬 Add your feedback	Jan 12, 2023 at 4:02PM (UTC)

9 Ignored Deployments

Name	Status	Updated
examples-basic-web	⬜️ Ignored (Inspect)	Jan 12, 2023 at 4:02PM (UTC)
examples-cra-web	⬜️ Ignored (Inspect)	Jan 12, 2023 at 4:02PM (UTC)
examples-designsystem-docs	⬜️ Ignored (Inspect)	Jan 12, 2023 at 4:02PM (UTC)
examples-kitchensink-blog	⬜️ Ignored (Inspect)	Jan 12, 2023 at 4:02PM (UTC)
examples-native-web	⬜️ Ignored (Inspect)	Jan 12, 2023 at 4:02PM (UTC)
examples-nonmonorepo	⬜️ Ignored (Inspect)	Jan 12, 2023 at 4:02PM (UTC)
examples-svelte-web	⬜️ Ignored (Inspect)	Jan 12, 2023 at 4:02PM (UTC)
examples-tailwind-web	⬜️ Ignored (Inspect)	Jan 12, 2023 at 4:02PM (UTC)
turbo-vite-web	⬜️ Ignored (Inspect)	Jan 12, 2023 at 4:02PM (UTC)

github-actions · 2023-01-12T12:51:49Z

Benchmark for `fd8b959`

Click to view benchmark

Test	Base	PR	%
bench_hmr_to_commit/Turbopack CSR/1000 modules	8653.96µs ± 46.17µs	8620.86µs ± 41.99µs	-0.38%
bench_hmr_to_commit/Turbopack RCC/1000 modules	8918.28µs ± 65.97µs	8977.85µs ± 42.91µs	+0.67%
bench_hmr_to_commit/Turbopack RSC/1000 modules	474.49ms ± 2.83ms	472.36ms ± 1.88ms	-0.45%
bench_hmr_to_commit/Turbopack SSR/1000 modules	8720.86µs ± 47.14µs	8694.98µs ± 45.84µs	-0.30%
bench_hmr_to_eval/Turbopack CSR/1000 modules	7623.01µs ± 72.45µs	7687.43µs ± 48.99µs	+0.85%
bench_hmr_to_eval/Turbopack RCC/1000 modules	7798.03µs ± 53.37µs	7794.52µs ± 50.73µs	-0.05%
bench_hmr_to_eval/Turbopack SSR/1000 modules	7653.41µs ± 64.93µs	7745.28µs ± 41.25µs	+1.20%
bench_hydration/Turbopack RCC/1000 modules	3339.05ms ± 8.17ms	3353.19ms ± 11.83ms	+0.42%
bench_hydration/Turbopack RSC/1000 modules	2842.97ms ± 9.54ms	2858.91ms ± 11.43ms	+0.56%
bench_hydration/Turbopack SSR/1000 modules	2654.13ms ± 10.13ms	2649.45ms ± 8.84ms	-0.18%
bench_startup/Turbopack CSR/1000 modules	1654.42ms ± 6.13ms	1636.82ms ± 6.45ms	-1.06%
bench_startup/Turbopack RCC/1000 modules	2501.11ms ± 8.82ms	2509.27ms ± 7.87ms	+0.33%
bench_startup/Turbopack RSC/1000 modules	2381.04ms ± 10.38ms	2398.48ms ± 6.87ms	+0.73%
bench_startup/Turbopack SSR/1000 modules	2049.14ms ± 4.71ms	2052.14ms ± 6.02ms	+0.15%

github-actions · 2023-01-12T12:57:00Z

🟢 CI successful 🟢

Thanks

alexkirsz

A few nits and questions, but otherwise :ship-it:

alexkirsz · 2023-01-12T13:18:06Z

crates/next-core/src/next_config.rs

+        let Some(experimental) = this.experimental.as_ref() else {
+            return Ok(WebpackLoadersOptionsVc::cell(WebpackLoadersOptions::default()));
+        };
+        let Some(turbopack_webpack_loaders) = experimental.turbopack_webpack_loaders.as_ref() else {
+            return Ok(WebpackLoadersOptionsVc::cell(WebpackLoadersOptions::default()));
+        };


Suggested change

let Some(experimental) = this.experimental.as_ref() else {

return Ok(WebpackLoadersOptionsVc::cell(WebpackLoadersOptions::default()));

};

let Some(turbopack_webpack_loaders) = experimental.turbopack_webpack_loaders.as_ref() else {

return Ok(WebpackLoadersOptionsVc::cell(WebpackLoadersOptions::default()));

};

let Some(turbopack_webpack_loaders) = this.experimental.as_ref().and_then(|experimental| experimental.turbopack_webpack_loaders.as_ref()) else {

return Ok(WebpackLoadersOptionsVc::cell(WebpackLoadersOptions::default()));

};

alexkirsz · 2023-01-12T13:20:56Z

crates/turbopack/src/module_options/module_options_context.rs

@@ -22,6 +44,7 @@ pub struct ModuleOptionsContext {
    pub enable_styled_components: bool,
    pub enable_styled_jsx: bool,
    pub enable_postcss_transform: Option<PostCssTransformOptions>,
+    pub enable_webpack_loaders: Option<WebpackLoadersOptions>,


It seems weird that this is worded as a boolean (enable_something) instead of just webpack_loaders, like we have for other options. Same comment goes for PostCSS.

yeah. pub enable_styled_jsx: bool, could also be pub styled_jsx: bool,

Let's do that in a follow-up PR

alexkirsz · 2023-01-12T13:22:57Z

crates/turbopack-core/src/server_fs.rs

    #[turbo_tasks::function]
    fn read(&self, _fs_path: FileSystemPathVc) -> Result<FileContentVc> {
-        bail!("Reading is not possible from the virtual filesystem for the dev server")
+        bail!("Reading is not possible from the marker filesystem for the server")


What is a "marker" file system?

We only used it for paths and do not read/write from it. A virtual filesystem would allow to read/write...

alexkirsz · 2023-01-12T13:24:17Z

crates/turbopack-node/js/package.json

+    "check": "tsc --noEmit"
+  },
+  "dependencies": {
+    "loader-runner": "^4.3.0"


How do we ensure this is installed by the user? Is it one of Next.js' transitive dependencies? Or should we compile it and distribute it with turbopack?

Sadly it's not accessible from next/dist. We probably should exposed that as compiled packages. It's currently bundled into webpack

Currently the user need to install it to use this feature. That's fine for the first iteration...

alexkirsz · 2023-01-12T13:27:23Z

crates/turbopack-node/js/tsconfig.json

+
+    // environment
+    "jsx": "react-jsx",
+    "lib": ["ESNext", "DOM"],


Does it make sense to include DOM type definitions for this package?

No. I just copied the tsconfig from next-dev^^

alexkirsz · 2023-01-12T13:29:17Z

crates/turbopack-node/src/transforms/webpack.rs

+}
+
+#[turbo_tasks::value]
+struct WebpackLoadersedAsset {


Suggested change

struct WebpackLoadersedAsset {

struct WebpackLoadersProcessedAsset {

Brooooooklyn · 2023-01-12T15:42:58Z

It looks like it would be easier to implement this with napi-rs, @sokra do you mind if I make a prototype based on this with napi-rs?

sokra · 2023-01-12T15:52:03Z

It looks like it would be easier to implement this with napi-rs, @sokra do you mind if I make a prototype based on this with napi-rs?

A few questions?

Can we use napi-rs even if you run the standalone binary next-dev.exe without being called from next.js?`
Can we create a separate Node.js Isolate with napi-rs that can be restarted separately?
Can we provide the JS source files from in-memory without them being written to disk at all?

Brooooooklyn · 2023-01-12T16:05:33Z

Can we use napi-rs even if you run the standalone binary next-dev.exe without being called from next.js?`

Why do we need to distribute turbopack as a binary? IMO the entry file can be a JavaScript file that calls the napi-rs wrapped Rust function like next-swc.

Can we create a separate Node.js Isolate with napi-rs that can be restarted separately?

We can use worker_threads API to do that, and I think if we can call JavaScript function directly in Rust, we can avoid restarting Node.js process scenario in most cases; just re-call the JavaScript function is fine.

Can we provide the JS source files from in-memory without them being written to disk at all?

Yes, sure; we can keep all the ability in turbopack with napi-rs, including the memory fs, resolve alias, and even dev-server.
Building turbopack with napi-rs just changes the way to distribute turbopack, we will not lose any features and abilities in turbopack.

github-actions · 2023-01-12T17:03:33Z

Benchmark for `e448d45`

Click to view benchmark

Test	Base	PR	%
bench_hmr_to_commit/Turbopack CSR/1000 modules	8058.07µs ± 39.91µs	8148.72µs ± 35.72µs	+1.12%
bench_hmr_to_commit/Turbopack RCC/1000 modules	8382.36µs ± 51.08µs	8399.79µs ± 47.82µs	+0.21%
bench_hmr_to_commit/Turbopack RSC/1000 modules	466.39ms ± 1.27ms	466.01ms ± 1.68ms	-0.08%
bench_hmr_to_commit/Turbopack SSR/1000 modules	8257.27µs ± 34.50µs	8143.01µs ± 58.31µs	-1.38%
bench_hmr_to_eval/Turbopack CSR/1000 modules	7160.96µs ± 66.89µs	7207.18µs ± 50.13µs	+0.65%
bench_hmr_to_eval/Turbopack RCC/1000 modules	7342.99µs ± 44.93µs	7345.22µs ± 55.73µs	+0.03%
bench_hmr_to_eval/Turbopack SSR/1000 modules	7228.54µs ± 62.71µs	7368.68µs ± 52.92µs	+1.94%
bench_hydration/Turbopack RCC/1000 modules	3300.49ms ± 10.53ms	3278.94ms ± 11.82ms	-0.65%
bench_hydration/Turbopack RSC/1000 modules	2806.19ms ± 8.70ms	2802.55ms ± 10.46ms	-0.13%
bench_hydration/Turbopack SSR/1000 modules	2596.12ms ± 7.92ms	2613.78ms ± 9.99ms	+0.68%
bench_startup/Turbopack CSR/1000 modules	1614.33ms ± 3.05ms	1614.55ms ± 6.48ms	+0.01%
bench_startup/Turbopack RCC/1000 modules	2448.58ms ± 8.18ms	2448.79ms ± 5.01ms	+0.01%
bench_startup/Turbopack RSC/1000 modules	2344.72ms ± 11.81ms	2371.13ms ± 7.48ms	+1.13%
bench_startup/Turbopack SSR/1000 modules	2016.93ms ± 2.42ms	2012.42ms ± 4.87ms	-0.22%

jridgewell · 2023-01-12T21:55:07Z

crates/turbopack-node/src/transforms/util.rs

+        .map(|(file, (content, _source_map))| {
+            // TODO handle SourceMap
+            VirtualAssetVc::new(
+                ServerFileSystemVc::new().root().join(&file),


Should we be passing in the file system to use?

ForsakenHarmony · 2023-01-16T13:06:35Z

crates/turbopack-node/src/transforms/webpack.rs

+fn webpack_loaders_executor(project_root: FileSystemPathVc, context: AssetContextVc) -> AssetVc {
+    EcmascriptModuleAssetVc::new(
+        VirtualAssetVc::new(
+            project_root.join("__turbopack__/webpack-loaders-executor.ts"),


Shouldn't that be [embedded_modules]/@vercel/turbopack-node

add the minimum to run a simple raw-loader (see test case) * add `experimental.turbopackWebpackLoaders` to next.config.js * key is extension like `".mdx"`, value is an array of loaders like `["mdx-loader"]`

sokra requested review from a team as code owners January 12, 2023 11:49

sokra requested review from mehulkar and chris-olszewski January 12, 2023 11:49

sokra requested a review from wbinnssmith January 12, 2023 11:49

github-actions bot added the area: turbopack label Jan 12, 2023

alexkirsz approved these changes Jan 12, 2023

View reviewed changes

add basic webpack loader support

3249c77

nitpicks

824f34b

sokra force-pushed the sokra/web-311-webpack-loaders branch from bed6138 to 824f34b Compare January 12, 2023 16:00

vercel bot deployed to Preview – turbo-site January 12, 2023 16:02 View deployment

wbinnssmith approved these changes Jan 12, 2023

View reviewed changes

sokra merged commit 160dd1b into main Jan 12, 2023

sokra deleted the sokra/web-311-webpack-loaders branch January 12, 2023 20:07

jridgewell reviewed Jan 12, 2023

View reviewed changes

ForsakenHarmony reviewed Jan 16, 2023

View reviewed changes

This was referenced Jan 20, 2023

build(cargo): update turbopack vercel/next.js#45108

Closed

build(cargo): update turbopack vercel/next.js#45109

Merged

DuCanhGH mentioned this pull request Mar 15, 2023

Fix(typedRoutes): fixed files from directories other than pages and app getting into Route vercel/next.js#47184

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add basic webpack loader support #3284

add basic webpack loader support #3284

sokra commented Jan 12, 2023

vercel bot commented Jan 12, 2023 •

edited

Loading

github-actions bot commented Jan 12, 2023

github-actions bot commented Jan 12, 2023 •

edited

Loading

alexkirsz left a comment

alexkirsz Jan 12, 2023

alexkirsz Jan 12, 2023

sokra Jan 12, 2023

sokra Jan 12, 2023

alexkirsz Jan 12, 2023

sokra Jan 12, 2023

alexkirsz Jan 12, 2023

sokra Jan 12, 2023

sokra Jan 12, 2023

alexkirsz Jan 12, 2023

sokra Jan 12, 2023

alexkirsz Jan 12, 2023

Brooooooklyn commented Jan 12, 2023

sokra commented Jan 12, 2023

Brooooooklyn commented Jan 12, 2023

github-actions bot commented Jan 12, 2023

jridgewell Jan 12, 2023

ForsakenHarmony Jan 16, 2023

	struct WebpackLoadersedAsset {
	struct WebpackLoadersProcessedAsset {

add basic webpack loader support #3284

add basic webpack loader support #3284

Conversation

sokra commented Jan 12, 2023

vercel bot commented Jan 12, 2023 • edited Loading

github-actions bot commented Jan 12, 2023

Benchmark for fd8b959

github-actions bot commented Jan 12, 2023 • edited Loading

🟢 CI successful 🟢

alexkirsz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Brooooooklyn commented Jan 12, 2023

sokra commented Jan 12, 2023

Brooooooklyn commented Jan 12, 2023

github-actions bot commented Jan 12, 2023

Benchmark for e448d45

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vercel bot commented Jan 12, 2023 •

edited

Loading

Benchmark for `fd8b959`

github-actions bot commented Jan 12, 2023 •

edited

Loading

Benchmark for `e448d45`