How do data segments interact with dynamic linking?

[titzer]

Forked discussion from #301

How can data segments (which specify the initial contents of memory at startup) be useful and integrate with dynamic linking? Should programs be able to dynamically load data segments into memory?

[kg]

If you can load a module at runtime, that implies that you can load a data segment at runtime - unless we prohibit non-main-modules from having data segments. I think that prohibition would be a difficult sell, though.

This implies some sort of address space management. If we want to do it in user space, we should expose a single primitive that user code can use to request that a module's data segment be mapped (loaded) into the heap at offset X.

[lukewagner]

Definitely agreed that shared modules should have global data sections; it's a basic need. Copying from comment in #301 to hopefully continue this line of discussion:

A main module knows it has the whole [0, memory_size) range to itself and can put anything anywhere infallibly (an invariant that could be leveraged for interesting optimizations). For a shared module, since wasm semantics don't say what memory in [0, memory_size) is already in use, I've been assuming that we'd want to specify an allocate_global_data_section(length) callback that is specified to be called by the engine when loading a shared module and allows the app to decide exactly how it wants to lay out global data. FWIW, the same issue comes up with aliased thread-local state (which needs to go in the heap) and could have the same callback solution. (There's a lot of symmetry between thread-local state and dynamically linked global state.)

[jfbastien]

As discussed in the PR, it would be nice if the basic user-side allocator (potentially provided by the toolchain) didn't have to magically know about where the module's data is loaded (because it wasn't asked "where would you want this?").

Furthermore, it would be nice if this basic allocator provided ASLR by default. Developers could still do whatever they want, but wouldn't need to reinvent ASLR since they'd get it by default. As things stand that would require copying the data section to a new location on load, which seems silly.

[lukewagner]

@jfbastien The toolchain gets to place the data so I don't understand "magically know"; the data is where the toolchain put the data.

You do make an interesting point that, if we require main module data segments to be loaded at a constant address, this would require doing something suboptimal for a toolchain that wanted to provide ASLR. Now, for shared modules, if we have the above-proposed allocator hook, the hook could easily do the randomization. This would suggest doing the same thing for the main module. The only quirk is that the main module's allocator hook would run with a zero'd heap (the heap was correctly-sized, but not yet initialized) so the hook's code would have to be written not to depend on any global heap state; but that's fine if all the hook wants to do is call random. This would provide more symmetry between main and shared modules, so I'm in favor.

[lskyum]

Just throwing something into the discussion here...

There could be cases where a wasm module is loaded into many browser tabs, for example a .NET/Java VM or some other popular component. Such modules will often have a bunch of static data that will be dublicated between the browser tabs - unless there is a mechanism to share that data without violating security. (For example by making it read-only.)

An example of static data could be language and encoding data in a .NET runtime.

[lukewagner]

@lskyum I agree that's an important scenario. In such cases, we'll also want browser caching to share machine code. The cache file won't be a literal wasm file, but something browser internal (or perhaps a native executable format like ELF). Given this, it seems like we could achieve sharing of the underlying pages by COW-mmap'ing the data segments from the cache file. This does rely on the destination address being a multiple of page size; we could consider mandating this (as already proposed for memory resizing).

[jfbastien]

@jfbastien The toolchain gets to place the data so I don't understand "magically know"; the data is where the toolchain put the data.

@lukewagner Seems like we agree on the overall idea, but I just wanted to clarify this: I just meant that the allocator's code will have to "know" about some magical extern globals for main module start and end location. Something akin to extern const void *__main_module_start, *__main_module_end;. It's not the end of the world, but it's pretty ugly compared to just asking the base allocator (enable ASLR and other things), though indeed the code then has to be able to run without an initialized heap!

[lukewagner]

@jfbastien Ah, I see. There is actually a small design question here: is there one allocate_global_data_section callback defined by the main module and called for all subsequent shared module loads or does each shared module define its own allocate_global_data_section callback? If we did the former, then everything happens in the main module which has all the knowledge and can integrate with the malloc/mmap impls. The latter sounds vaguely more powerful and encapsulated, but it needs the magic imports you were talking about (which would just be part of the ABI that we need to define anyway). Either strategy could be mapped to the other via ABI convention.

[jfbastien]

I think we want to have the following layering:

1. Developers can use any regular malloc they want.
   • These mallocs work as-is (no code change if they're compiled for a POSIX-y system!).
   • These mallocs use mmap.
   • There's one malloc per wasm module (dsos use the main module's allocator).
2. The toolchain provides a global allocator named mmap which serves a shim between malloc and wasm's magical innards.
   • This is specific to the wasm platform, but developers can roll their own.
   • There's again just one such function per wasm module.
   • This function is known to the wasm platform, and is called by the loader to perform initialization of the heap (including allocating data sections, and when loading dsos). It effectively keeps its own "page table" of the internal heap.
3. The wasm platform has magical syscalls for heap allocation and growth.
   • These call into the trusted runtime.

[lukewagner]

@jfbastien Sounds right to me.

[AndrewScheidecker]

I think what @jfbastien said works is about as good as you can do within the current memory model, but to me it seems inevitable that wasm must abandon the idea that a module has absolute control over its address space. I think a model closer to OS processes can be efficiently polyfilled and will more easily accommodate dynamic linking.

Some things we can take from OSes:

• Processes as isolated address spaces.
• Modules don't get their own address space, or any isolation from other modules running in the same process, or from the OS.
• The OS manages the address space of the process, and the module must go through it for page-level allocations.

Some things we shouldn't take from OSes:

• The idea of modules being loaded at static addresses.
• You only have control over the direct dependencies of your module, making it easy to get conflicts in indirect dependencies. I think that can be solved by a process-wide dictionary from semantic names (e.g. libc v34) to URIs, but I don't want to derail this thread with that half-baked idea.

If we applied these ideas to WebAssembly:

• We can define a wasm process with an address space isolated from other processes. Make minimal guarantees about the address space: maybe just a minimum size and a protected first page.
• Define a platform API for allocating virtual address space and committing physical pages to it. mmap(pointer?,I64) -> pointer and munmap(pointer,I64) -> bool are all that's really needed.
• The platform loader needs to cooperate with the map/unmap API to reserve address space for module data segments.

This could all be implemented efficiently in the polyfill as a page allocator for the asm.js linear memory. The polyfill could also require the application to provide an initial size for linear memory, a hint that would no longer be necessary for the wasm module itself.

I expect native implementations would reserve a fixed range of addresses for a wasm process, and generate memory access code using an immutable base and bounds check. However you do it, supporting a true 64-bit address space would likely require a separate OS process for each wasm process, which has its own implications for things like APIs to talk to the browser.

Thoughts?

[sunfishcode]

@AndrewScheidecker This issue here is just about working out the details of how to load data segments into memory within the current design. What you're proposing sounds like a pretty major change to the design. Would you mind filing a new issue so that it can get top-level visibility and discussion, rather than being buried here in the comments for this issue? Thanks!

[lukewagner]

@AndrewScheidecker Actually, what we're calling a "wasm module" is almost exactly the wasm process you described. For background, read discussion in #285 (especially about the contiguity issue) and the resulting FAQ. If you still have specific suggestions, feel free to open a new issue as @sunfishcode suggested. [Edit: oops, I didn't see you already did create #306]

[lskyum]

@lukewagner I really like the idea of sharing static memory through memory mapping. (with or without COW.) I think it's important that the static memory can be available from initial startup and doesn't have to be downloaded asynchronously.

I have been thinking about memory mapping for passing values for JS-interop, instead of having to copy everything. For example if a wasm application uses a lot of AJAX.

[AndrewScheidecker]

@lukewagner Actually, what we're calling a "wasm module" is almost exactly the wasm process you described. For background, read discussion in #285 (especially about the contiguity issue) and the resulting FAQ.

I think we're on the same page (hah hah) about the contiguity issue. Software page protection would be an onerous requirement, so WASM processes should have an contiguous address space to work in that can be protected by simple bounds checks. I'm just saying that they should have to ask the VM to commit pages within the address space, rather than starting the address space 100% committed and forcing the wasm loader to ask the wasm process where to put dynamically loaded module data sections.

I also understand that the spec's "wasm module" encompasses both a unit of code (as described at https://github.com/WebAssembly/design/blob/master/Modules.md), and an execution context that may include multiple of such units that share memory. I find that confusing, which is why I was using the term process in place of the latter use of module. To summarize part of the discussion in #285 using this terminology, it's useful to have multiple wasm modules running in a single process, and a single browser tab may include multiple processes from different sources that don't trust each other.

@lskyum I have been thinking about memory mapping for passing values for JS-interop, instead of having to copy everything. For example if a wasm application uses a lot of AJAX.

You might find #304 interesting.

[sunfishcode]

Pulling this into MVP, which now has dynamic-linking mechanisms, and the answer to the above question within the context of #682 is:

Yes, linking in a module causes it to load its "segments" into the linear address space, and the offsets where it loads them are "initializer expressions" which can reference imported global variables, which tell them what offset to load their segments at. These global variables can be initialized to values passed in as arguments to the Wasm.instantiateModule call, so each library can be instantiated with its own base address to load its segment at.