[Wasm] JS modularization #61313

pavelsavara · 2021-11-08T14:36:02Z

Modularized dotnet.js

By using emcc options MODULARIZE and EXPORT_ES6, depending on WasmEnableES6 to produce ES6 or CommonJS modules respectively.
WasmEnableES6 enables WasmBuildNative because dotnet.js need to be re-linked
CommonJS version is able to be loaded into global namespace and behaves as before this change.
Added new es6/*.js and es6/*.js files which are included in dotnet.js creation. Details are described in modularize-dotnet.md
it exports createDotnetRuntime method instead of global Module
- for usage see src\mono\sample\wasm\browser-es6\runtime.js
- createDotnetRuntime(factory: (exports) => DotnetModuleConfig)
  takes function which creates Module-like config.
- createDotnetRuntime returns Promise<{MONO, INTERNAL, BINDING, Module}> exports.
New Module extension DotnetModuleConfig has following new fields
- disableDotNet6Compatibility - when true and if runtime script is loaded as module, it would not pollute global namespace.
- configSrc - location of mono-config.json as defined by MonoConfig .ts type
- onConfigLoaded - callback called early during runtime init, when MonoConfig was loaded but runtime didn't start yet.
- onDotnetReady - callback called when runtime is ready and all artifacts were loaded from MonoConfig. Or not called at all when onRuntimeInitialized is overriden in Blazor.
Key aspects are documented in src/mono/wasm/runtime/modularize-dotnet.md of this PR.

Other changes

Improved dotnet.d.ts it is now generated into version control too, so that we could observe how it evolves.
Removed legacy --testing argument and simplified is_testing logic in the the samples.
Added browser-es6 sample app, which uses WasmEnableES6=true to compile dotnet.js as ES6 module.
Added browser-legacy sample, which uses dotnet.js CommonJS module and loads it into global namespace.
Added package.json into the nupkg
Tested that Blazor works with CommonJS-global output
Drafted how Blazor could use the new API in [wasm] dotnet.js modularization aspnetcore#38721

Out of scope

switch to ES6 as default target
update of samples and unit tests to load ES6

NodeJs Usage after the change

import { createRequire } from 'module';
import createDotnetRuntime from 'dotnet'
import { dirname } from 'path';
import { fileURLToPath } from 'url';
import { exit } from 'process';

const { BINDING } = await createDotnetRuntime(() => ({
    imports: {
        require: createRequire(import.meta.url)
    },
    scriptDirectory: dirname(fileURLToPath(import.meta.url)) + '/',
    disableDotnet6Compatibility: true,
    configSrc: "./mono-config.json"
}));

try {
    const app_args = process.argv.slice(2);
    const result = await BINDING.call_assembly_entry_point("Wasm.Console.TypeScript.Sample.dll", [app_args], "m");
    exit(result)
} catch (error) {
    console.log("WASM ERROR " + error);
}

dotnet-issue-labeler · 2021-11-08T14:36:07Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

ghost · 2021-11-08T14:36:11Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

work in progress 7

Author:	pavelsavara
Assignees:	-
Labels:	`arch-wasm`
Milestone:	7.0.0

src/mono/wasm/runtime-test.js

src/mono/sample/wasm/browser-es6/Wasm.Browser.Sample.csproj

kg

Looks great, thanks for doing all this work! You don't need to make all the changes I've requested here

src/mono/sample/wasm/browser-es6/runtime.js

src/mono/sample/wasm/browser/runtime.js

src/mono/wasm/build/README.md

src/mono/wasm/build/WasmApp.Native.targets

src/mono/wasm/runtime/exports.ts

src/mono/wasm/runtime/modularize-dotnet.pre.js

src/mono/wasm/wasm.proj

src/mono/wasm/build/WasmApp.Native.targets

pavelsavara · 2021-11-09T14:02:49Z

This is a weird signature - do I understand it right and it's like this? createRuntime (callback : Function) : Promise<{MONO,INTERNAL,BINDING,Module}> and then callback is callback (namespaces : {MONO,INTERNAL,BINDING,Module}) : ConfigObject? How are the namespaces visible to the config getter if the runtime hasn't loaded yet? What will happen if it calls methods on them?

They are both receiving the same API object instances, i.e. same MONO instance.
In the callback MONO and others are empty objects and they become populated before the callbacks are called.
The return from createDotnetRuntime is obviously assigned only after the promise is resolved.

The callback could also be written with return {...} syntax, but adding statements before that return is not useful exactly because MONO and such are just empty shells.

You can think about it as if MONO was populated very early, during exports.ts __exportAPI function. The actual path is bit more complex.

The functionality in the onConfigLoaded is pretty limited as it's before full start of mono VM. It happens during emscripten preInit event, after fetched config.

The onDotNetReady is called after all assets are loaded , mapping to original config.loaded_cb. It happens during emscripten onRuntimeInitialized after monoVm init + globalization + assemblies. This also matches when the top level promise is resolved. The original emscripten Module.ready promise is replaced this this one.

pavelsavara · 2021-11-09T16:29:34Z

Currently this corrupts some of the xharness testResults.xml files. It seems to me it could be timing or something with console.

pavelsavara · 2021-11-09T19:27:56Z

I added src/mono/wasm/runtime/modularize-dotnet.md which is probably worth reading for this PR.

pavelsavara · 2021-11-09T23:05:58Z

test failure runtime.js 2:34 Uncaught SyntaxError: Unexpected reserved word helix

kg

Looks good to me, assuming no one else finds any problems!

pavelsavara · 2021-11-10T07:03:57Z

@radical please review the msbuild parts.

maraf

I only have a slight feeling that from user point of view, this syntax may seem a bit odd.

export const { MONO, BINDING } = await createDotnetRuntime(({ MONO, BINDING, Module }) => ...);

I mean "what is the difference between left MONO and right MONO?"

Anyway, I can't find a better solution.

radical

Reviewed the parts that I could

radical · 2021-11-10T09:05:52Z

Currently this corrupts some of the xharness testResults.xml files. It seems to me it could be timing or something with console.

The testResults.xml file is incomplete in those instances. WASM EXIT is the last line written, which indicates that the tests have run, and the xml file has been written too. I'm guessing that this is getting written too soon in the updated runtime-test.js.

radical · 2021-11-10T09:09:32Z

The AOT System.Runtime.Tests fail to run with (log):

[22:32:58] info: Running v8 --expose_wasm --stack-trace-limit=1000 runtime.js -- --run WasmTestRunner.dll System.Runtime.Tests.dll -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
                 
[22:32:58] fail: console.error: Error executing file
                 
[22:32:58] fail: console.error: failed to load the dotnet.js file
                 
[22:32:58] info: ./dotnet.js:3: SyntaxError: Cannot use 'import.meta' outside a module
[22:32:58] info:   var _scriptDir = import.meta.url;
[22:32:58] info:                           ^^^^
[22:32:58] info: SyntaxError: Cannot use 'import.meta' outside a module
[22:32:58] info:     at loadScript (runtime.js:361:24)
[22:32:58] info:     at loadDotnet (runtime.js:369:12)
[22:32:58] info:     at runtime.js:105:1
[22:32:58] info: 
[22:32:58] info: Incoming arguments: --run WasmTestRunner.dll System.Runtime.Tests.dll -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing
[22:32:58] info: Application arguments: --run WasmTestRunner.dll System.Runtime.Tests.dll -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing

Also, the app then incorrectly exits with exit code 0, thus it records as tests having passed. This should be fixed so if such an error occurs, then it returns a non-zero exit code.

radical · 2021-11-10T09:11:44Z

Also, make sure to run all the wasm tests. A lot of them are disabled now for PRs, and run fully only for rolling builds, generally conditioned with isFullMatrix in eng/pipelines/runtime.yml, and eng/pipelines/runtime-staging.yml.

And cc @thaystg to confirm that debugging still works, and the tests pass too.

radical

I reviewed the parts that I could (so, skipped js, and ts!). looking good.
Just some suggestions.

eng/liveBuilds.targets

src/installer/pkg/sfx/Microsoft.NETCore.App/Directory.Build.props

radical · 2021-12-01T05:31:59Z

src/mono/sample/mbr/browser/main.js

@@ -7,7 +7,7 @@ var Module = {
    onConfigLoaded: function () {
        MONO.config.environment_variables["DOTNET_MODIFIABLE_ASSEMBLIES"] = "debug";
    },
-    onDotNetReady: function () {
+    onDotnetReady: function () {


why the change? DotNet is the style used usually, including package names.

my quick survey told me Dotnet is more common, but I don't have strong opinion.
@lewing this would stick with us going forward, what should it be ?

I will eventually rename it in next PR

radical · 2021-12-01T05:35:48Z

src/mono/sample/wasm/browser-legacy/Wasm.Browser.LegacySample.csproj

+    <DebugSymbols>true</DebugSymbols>
+    <DebugType>embedded</DebugType>
+    <WasmDebugLevel>1</WasmDebugLevel>
+    <WasmBuildNative>true</WasmBuildNative>


We should remove it for this PR, then. And reverse the property setting here, and in the es6 sample, when es6 becomes default.

src/mono/sample/wasm/browser-legacy/Wasm.Browser.LegacySample.csproj

radical · 2021-12-01T05:51:52Z

src/mono/wasm/runtime/CMakeLists.txt

+    LINK_DEPENDS "${NATIVE_BIN_DIR}/src/emcc-default.rsp;${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.pre.js;${NATIVE_BIN_DIR}/src/cjs/runtime.cjs.iffe.js;${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.lib.js;${NATIVE_BIN_DIR}/src/pal_random.lib.js;${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.post.js;${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.extpost.js;"
+    LINK_FLAGS "@${NATIVE_BIN_DIR}/src/emcc-default.rsp ${CONFIGURATION_LINK_FLAGS} -DENABLE_NETCORE=1 --extern-pre-js ${NATIVE_BIN_DIR}/src/cjs/runtime.cjs.iffe.js --pre-js ${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.pre.js --js-library ${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.lib.js --js-library ${NATIVE_BIN_DIR}/src/pal_random.lib.js --post-js ${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.post.js --extern-post-js ${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.extpost.js "


The list here also should be extracted into variables. Doesn't have to be in this PR.

@radekdoulik how could I do this ? I'll do it in next PR.

src/mono/wasm/runtime/modularize-dotnet.md

src/mono/wasm/test-main.js

src/mono/wasm/wasm.proj

Co-authored-by: Ankit Jain <radical@gmail.com>

radical

Reviewed the bits that I could, and those look good to me. great job 👍

pavelsavara · 2021-12-01T06:40:20Z

/azp run runtime-manual

azure-pipelines · 2021-12-01T06:40:33Z

Azure Pipelines successfully started running 1 pipeline(s).

maraf

debugger-main.js still uses onDotNetReady (capital N)

src/mono/wasm/runtime/polyfills.ts

src/mono/wasm/runtime/types/emscripten.ts

src/mono/wasm/runtime/modularize-dotnet.md

src/mono/wasm/runtime/exports.ts

pavelsavara · 2021-12-01T15:07:50Z

/azp run runtime-manual

azure-pipelines · 2021-12-01T15:08:04Z

Azure Pipelines successfully started running 1 pipeline(s).

pavelsavara · 2021-12-01T16:47:14Z

/azp run runtime-manual

azure-pipelines · 2021-12-01T16:49:55Z

Azure Pipelines successfully started running 1 pipeline(s).

radical · 2021-12-01T20:05:32Z

Some of the wasm.build.tests have:

[yc2d32mb.kow]          EXEC : warning : invalid item (maybe a typo?) in EXPORTED_RUNTIME_METHODS: MONO [/datadisks/disk1/work/AAF1094C/w/BA0209FA/e/yc2d32mb.kow/rebuild_flags_Release.csproj]
[yc2d32mb.kow]          EXEC : warning : invalid item (maybe a typo?) in EXPORTED_RUNTIME_METHODS: BINDING [/datadisks/disk1/work/AAF1094C/w/BA0209FA/e/yc2d32mb.kow/rebuild_flags_Release.csproj]
[yc2d32mb.kow]          EXEC : warning : invalid item (maybe a typo?) in EXPORTED_RUNTIME_METHODS: INTERNAL [/datadisks/disk1/work/AAF1094C/w/BA0209FA/e/yc2d32mb.kow/rebuild_flags_Release.csproj]

pavelsavara · 2021-12-01T21:45:58Z

EXPORTED_RUNTIME_METHODS - I will fix it
I tested debugger unit tests, only well known DebuggerTests.HarnessTests.BrowserClose fails.
On CI [FAIL] System.Diagnostics.Tests.StopwatchTests.GetTimestamp is System.Diagnostics.Tests.StopwatchTests.GetTimestamp test fails in the CI #62021
On CI Assertion failed: native function malloc called before runtime initialization - is when emcc -s ASSERTIONS=1. Blazor has the same issue I think. It works anyway, when asserts are not enabled. I will change it back to run all in onRuntimeInitialized, but in that case postRun event is running in parallel with loading of assemblies and code execution. This would be consistent with previous runtime behavior.

pavelsavara · 2021-12-01T22:07:48Z

/azp run runtime-manual

azure-pipelines · 2021-12-01T22:08:04Z

Azure Pipelines successfully started running 1 pipeline(s).

pavelsavara added the arch-wasm WebAssembly architecture label Nov 8, 2021

pavelsavara added this to the 7.0.0 milestone Nov 8, 2021

pavelsavara requested review from radical, kg and maraf November 8, 2021 14:36

pavelsavara mentioned this pull request Nov 8, 2021

[wasm] Tests on CI: Full version of libmono-component-diagnostics_tracing-static.a is linked instead of stub #61294

Open

runfoapp bot mentioned this pull request Nov 8, 2021

RunContinueWithStressTestsNoState timing out in CI #2271

Closed

pavelsavara force-pushed the wasm_modularization7 branch from 4a9b019 to 9649380 Compare November 8, 2021 20:14

pavelsavara changed the title ~~DRAFT: [Wasm] JS modularization~~ [Wasm] JS modularization Nov 8, 2021

pavelsavara marked this pull request as ready for review November 8, 2021 20:21

pavelsavara requested a review from marek-safar as a code owner November 8, 2021 20:21

am11 reviewed Nov 8, 2021

View reviewed changes

src/mono/wasm/runtime-test.js Outdated Show resolved Hide resolved

kg reviewed Nov 8, 2021

View reviewed changes

src/mono/sample/wasm/browser-es6/Wasm.Browser.Sample.csproj Outdated Show resolved Hide resolved

kg requested changes Nov 8, 2021

View reviewed changes

radekdoulik reviewed Nov 9, 2021

View reviewed changes

src/mono/wasm/build/WasmApp.Native.targets Outdated Show resolved Hide resolved

pavelsavara force-pushed the wasm_modularization7 branch from 9649380 to a43160b Compare November 9, 2021 19:25

pavelsavara requested a review from kg November 9, 2021 19:28

kg approved these changes Nov 10, 2021

View reviewed changes

maraf approved these changes Nov 10, 2021

View reviewed changes

radical previously requested changes Nov 10, 2021

View reviewed changes

pavelsavara requested a review from kg December 1, 2021 05:01

radical requested changes Dec 1, 2021

View reviewed changes

ghost added the needs-author-action An issue or pull request that requires more info or actions from the author. label Dec 1, 2021

Update src/mono/wasm/runtime/modularize-dotnet.md

911bb5b

Co-authored-by: Ankit Jain <radical@gmail.com>

ghost removed the needs-author-action An issue or pull request that requires more info or actions from the author. label Dec 1, 2021

pavelsavara and others added 2 commits December 1, 2021 07:14

Update eng/liveBuilds.targets

072568d

Co-authored-by: Ankit Jain <radical@gmail.com>

feedback from @radical

c3df6fd

radical approved these changes Dec 1, 2021

View reviewed changes

maraf reviewed Dec 1, 2021

View reviewed changes

src/mono/wasm/runtime/polyfills.ts Show resolved Hide resolved

src/mono/wasm/runtime/types/emscripten.ts Outdated Show resolved Hide resolved

src/mono/wasm/runtime/modularize-dotnet.md Outdated Show resolved Hide resolved

src/mono/wasm/runtime/exports.ts Outdated Show resolved Hide resolved

pavelsavara added 2 commits December 1, 2021 15:56

@maraf's feedback

49fcd84

tweaks

00d784d

fix variables

ff39ea5

fix exports, fix early malloc

8ddc710

maraf approved these changes Dec 2, 2021

View reviewed changes

pavelsavara merged commit 1bd3630 into dotnet:main Dec 2, 2021

pavelsavara deleted the wasm_modularization7 branch December 2, 2021 09:42

This was referenced Dec 3, 2021

[wasm] Random tests failing with main.js 384:18 Uncaught Error: Can't load ./dotnet.js #62304

Closed

[WASM] Adding a new switch to run wasm on NodeJS #54640

Closed

elringus mentioned this pull request Dec 23, 2021

Migrate to stock .NET runtime elringus/bootsharp#20

Closed

ghost locked as resolved and limited conversation to collaborators Jan 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Wasm] JS modularization #61313

[Wasm] JS modularization #61313

pavelsavara commented Nov 8, 2021 •

edited

Loading

dotnet-issue-labeler bot commented Nov 8, 2021

ghost commented Nov 8, 2021

kg left a comment

pavelsavara commented Nov 9, 2021

pavelsavara commented Nov 9, 2021

pavelsavara commented Nov 9, 2021

pavelsavara commented Nov 9, 2021

kg left a comment

pavelsavara commented Nov 10, 2021

maraf left a comment

radical left a comment

radical commented Nov 10, 2021 •

edited

Loading

radical commented Nov 10, 2021

radical commented Nov 10, 2021

radical left a comment

radical Dec 1, 2021

pavelsavara Dec 1, 2021

pavelsavara Dec 1, 2021

radical Dec 1, 2021

radical Dec 1, 2021

pavelsavara Dec 1, 2021

radical left a comment •

edited

Loading

pavelsavara commented Dec 1, 2021

azure-pipelines bot commented Dec 1, 2021

maraf left a comment

pavelsavara commented Dec 1, 2021

azure-pipelines bot commented Dec 1, 2021

pavelsavara commented Dec 1, 2021

azure-pipelines bot commented Dec 1, 2021

radical commented Dec 1, 2021

pavelsavara commented Dec 1, 2021

pavelsavara commented Dec 1, 2021

azure-pipelines bot commented Dec 1, 2021

		LINK_DEPENDS "${NATIVE_BIN_DIR}/src/emcc-default.rsp;${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.pre.js;${NATIVE_BIN_DIR}/src/cjs/runtime.cjs.iffe.js;${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.lib.js;${NATIVE_BIN_DIR}/src/pal_random.lib.js;${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.post.js;${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.extpost.js;"
		LINK_FLAGS "@${NATIVE_BIN_DIR}/src/emcc-default.rsp ${CONFIGURATION_LINK_FLAGS} -DENABLE_NETCORE=1 --extern-pre-js ${NATIVE_BIN_DIR}/src/cjs/runtime.cjs.iffe.js --pre-js ${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.pre.js --js-library ${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.lib.js --js-library ${NATIVE_BIN_DIR}/src/pal_random.lib.js --post-js ${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.post.js --extern-post-js ${NATIVE_BIN_DIR}/src/cjs/dotnet.cjs.extpost.js "

[Wasm] JS modularization #61313

[Wasm] JS modularization #61313

Conversation

pavelsavara commented Nov 8, 2021 • edited Loading

Modularized dotnet.js

Other changes

Out of scope

NodeJs Usage after the change

dotnet-issue-labeler bot commented Nov 8, 2021

ghost commented Nov 8, 2021

kg left a comment

Choose a reason for hiding this comment

pavelsavara commented Nov 9, 2021

pavelsavara commented Nov 9, 2021

pavelsavara commented Nov 9, 2021

pavelsavara commented Nov 9, 2021

kg left a comment

Choose a reason for hiding this comment

pavelsavara commented Nov 10, 2021

maraf left a comment

Choose a reason for hiding this comment

radical left a comment

Choose a reason for hiding this comment

radical commented Nov 10, 2021 • edited Loading

radical commented Nov 10, 2021

radical commented Nov 10, 2021

radical left a comment

Choose a reason for hiding this comment

radical Dec 1, 2021

Choose a reason for hiding this comment

pavelsavara Dec 1, 2021

Choose a reason for hiding this comment

pavelsavara Dec 1, 2021

Choose a reason for hiding this comment

radical Dec 1, 2021

Choose a reason for hiding this comment

radical Dec 1, 2021

Choose a reason for hiding this comment

pavelsavara Dec 1, 2021

Choose a reason for hiding this comment

radical left a comment • edited Loading

Choose a reason for hiding this comment

pavelsavara commented Dec 1, 2021

azure-pipelines bot commented Dec 1, 2021

maraf left a comment

Choose a reason for hiding this comment

pavelsavara commented Dec 1, 2021

azure-pipelines bot commented Dec 1, 2021

pavelsavara commented Dec 1, 2021

azure-pipelines bot commented Dec 1, 2021

radical commented Dec 1, 2021

pavelsavara commented Dec 1, 2021

pavelsavara commented Dec 1, 2021

azure-pipelines bot commented Dec 1, 2021

pavelsavara commented Nov 8, 2021 •

edited

Loading

radical commented Nov 10, 2021 •

edited

Loading

radical left a comment •

edited

Loading