qjsc-like compiler

[reaktivo]

Would it be possible to have a qjsc-like compiler built into txiki that would allow bundling user code with the txiki runtime?

[saghul]

Yeah. Ideally txikijs would do it itself, with some -c flag for example...

[ArtDmn]

Is it still something investigated ?

[saghul]

It is something I'd like to eventually do yeah.

[Tobbe]

One more vote for this!

Wrote up a whole new issue before I saw this one was already literally at the top of the list 🤦 🙈

---

Hi. Very cool project you have here! 🙂

I'm looking for a way to write a .js file and then wrap it up together with a JS engine into a single-file executable application that works on Linux, Windows and MacOS

NodeJS supports this with the --experimental-sea-config flag and a bunch more steps
deno supports it with deno compile
bun supports it with bun build ./cli.ts --compile --outfile mycli

Those all produce executables in the 50-100 mb range for a simple console.log('Hello World') though. I was hoping to build something smaller 🤏

Would it be possible to do something like this with txiki.js?

[Tobbe]

If I start writing something for NodeJS or Bun today, is there something I should watch out for, that would make it difficult/impossible to then later move it to txiki.js? When/if this eventually gets implemented here, of course.

[saghul]

Not sure I understand the question :-)

If you write a script without any runtime-specific API it should, in principle, work across all of them.

[Tobbe]

If you write a script without any runtime-specific API it should, in principle, work across all of them.

When you wrote this I didn't really understand what implications this had. Now, with the discussions in #525 and #515, I realize just how big of a hurdle this would be to get things I'm used to writing working with txiki.js 😅 But that definitely does not mean I'm deterred from trying to use txiki.js for what I initially wanted to do with it. If anything I'm even more excited to try to see what I can build as soon as you have some time to spend on tjs compile functionality 😍

[MulverineX]

Ideally, this should also have an option to compile to non-executable bytecode (not bundling the runtime).

[saghul]

Indeed.

[saghul]

I was looking into this just today, and I think there are 2 ways it can go:

Option A:

tjs compile file.js

That would bytecompile file.js and use a similat trick to Deno: create a new executable which is a copy of the current one, then add the bytecode, an index and a trailer. When the self-contained executable starts, it will check if it ends in the trailer and if so, get the index, and evaluate the bundled bytecode.

The cons of this is that there needs to be a bundling process, in order to bytecompile the self-contained executable without external dependencies in imports and the like. Dynamic imports might be challening.

Option B:

tjs compile directory

The directory must contain an index.js file.

Here we'd (ab)use the ZIP file format. The trick is to compress the directory, and simply append the resulting ZIP file to a copy of the current executable.

Then when the executable is launched, it will check if it embeds a ZIP file, if it does, extract and run index.js .

Since we control the import functions we can make it so files are read from the ZIP file when importa happen, so no bundling is necessary!

This can be implemented with miniz which I want to add anyway and expose it as an API.

Thoughts?

Credits for option B: https://github.com/creationix/seaduk/blob/99da0b7d9acf59b851e9aacfca2e20e26c865d7c/src/main.c#L399 (it was buried somewhere in my memory...)

[MulverineX]

I'm pretty sure option A is what's desired here. Avoiding the parsing step entirely is the goal.

[saghul]

Care to elaborate? I lean more towards B, because it's more flexible and avoids having to always run a bundler, but I might be missing something.

I never thought the goal was to avoid parsing, but to have an easy way to create single-purpose executables.

[MulverineX]

It's really quite wasteful to do the parsing/interpreting step every time the executable is ran, especially on end-user machines.

This is why I suggested there also be an option to just compile the bytecode, not an executable. This way is even more optimized, since for many compiled bytecode scripts you only need one qjs runtime.

[saghul]

It's really quite wasteful to do the parsing/interpreting step every time the executable is ran, especially on end-user machines.

Every interpreter I know, except Python with the pyc files does this already.

I just rememred the main reason for not outputting bytecode directly, at least as a first step: it's not backwards compatible. Bytecode generated with a given version of qjsc need not be compatible with a newer or older version.

That is a different rabbithole.

[Tobbe]

For what it's worth, personally I'm more concerned about executable/bundle size than execution speed.

[saghul]

I think bundle size won't be vastly different across options, since we can strip the generated bytecode, to save some space. I guess using the ZIP mechanism might have some space savings too.

[ArtDmn]

I think a certain amount of people are willing to compile the code in order to obfuscate its source. I don't know if the zip solution would handle that, and I don't know either the proportion of people that want to compile mainly for this reason, regardless of size.

[saghul]

Good point. THe bytecode generation can strip the source for example.

[MulverineX]

it's not backwards compatible. Bytecode generated with a given version of qjsc need not be compatible with a newer or older version.

🤷 As long as you maintain a functional versioning scheme this isn't really an issue.

[saghul]

I was looking into this just today, and I think there are 2 ways it can go:

Option A:

tjs compile file.js

That would bytecompile file.js and use a similat trick to Deno: create a new executable which is a copy of the current one, then add the bytecode, an index and a trailer. When the self-contained executable starts, it will check if it ends in the trailer and if so, get the index, and evaluate the bundled bytecode.

The cons of this is that there needs to be a bundling process, in order to bytecompile the self-contained executable without external dependencies in imports and the like. Dynamic imports might be challening.

Option B:

tjs compile directory

The directory must contain an index.js file.

Here we'd (ab)use the ZIP file format. The trick is to compress the directory, and simply append the resulting ZIP file to a copy of the current executable.

Then when the executable is launched, it will check if it embeds a ZIP file, if it does, extract and run index.js .

Since we control the import functions we can make it so files are read from the ZIP file when importa happen, so no bundling is necessary!

This can be implemented with miniz which I want to add anyway and expose it as an API.

Thoughts?

Credits for option B: https://github.com/creationix/seaduk/blob/99da0b7d9acf59b851e9aacfca2e20e26c865d7c/src/main.c#L399 (it was buried somewhere in my memory...)

Option A just landed in master :-) Let's see how that does!