External modules & optional core features

[KaruroChori]

Case

At the moment txiki.js is a monolithic distribution.
This poses several issues while integrating it in complete applications.

Avoiding forking

When integrating txiki.js in our application, we cannot disable core features which are not needed (like the sizable WebAssembly module) or adding new ones, unless we want to maintain a divergent branch.
The ffi module is great, has good ergonomics and shifts a lot of problems on the application side; still, its overhead is not always acceptable and there are cases in which a deeper integration of custom libraries with the runtime is the better choice; no less as they are compiled in faster bytecode via qjsc before bundling.
This proposal would avoid the need for most of the divergent branches, ensuring the integrator with a greater degree of flexibility, and minimal impact on the current base-code & build process.

Platform specific modules & minimalism

As a monolithic distribution, there is some care needed when selecting features to avoid creep and ensure minimalism of the core runtime.
This can come at the expense of relevant but platform-specific modules which cannot be reasonably shipped as part of the main project; or, at the very least, they should not be enabled by default for everyone.
To provide some realistic examples:

• the direct support of GPIO pins for many of the ARM single board computer
• a react-like LVGL integration which would be great to quickly design GUIs in embedded systems.

Otherwise, we end up with a graveyard of side-projects using txiki.js like this. It manually embeds a now old version of txiki.js, and trying to fix this now is virtually impossible without a major refactoring.

Proposal

My suggestion is:

• to introduce extras sub-folders not tracked in git, where additional dependencies and core code can be added to the project (basically we would have to add those entries to .gitignore once for all). They would have to cover deps, benchmark, examples, tests, src/bundles/c (shared across all extra modules) & src/
• to add a not tracked modules.toml (or yaml/json, whatever) file to list all desired packages which are going to be downloaded and expanded in the relevant folder structure.
• A simple build script recalled by make to download these external dependencies, unpack them in the new extra folders, and ensure they end up in the next build steps.
• (optional) flags in the same modules.toml file as before where core features can be flagged.

I would like to stress that this is not meant to be a full packet manager, nor it should be.
All we need are:

• a few minor changes to simplify the integration of custom code while keeping txiki.js on the master branch
• a utility script to integrate in make to simplify the process of retrieving optional modules from local disk or remote repositories

Sure, we can perform some very basic checks, like matching a specific version of the runtime (or a range of it), but this is as far as I would go.
Realistically we don't want and cannot ship multiple versions of the same module as core due to aliasing, so handling parenting, cyclic deps etc is not really something we should care to support.

External ts types

They are to be handled as separate npm packages. This is in line with what is done for the current core as well; as such their distribution is out of scope in respect to this proposal.

Affected issues

Implementing this proposal would offer some relief for #160.
It is not a full solution, but we can easily structure most of our application to be an external module, ensuring most of its code will be handled as bytecode, while keeping small bits and configuration as normal code.

[saghul]

I like your proposal. I thought you were going to suggest the usual expose all the headers and so on, but it's more sensible, I like it.

We can iron out the details, but I like that you are thinking about build time!

Implementing this proposal would offer some relief for #160.

I'm not so sure. I could easily add a tjs compile foo.js thing right now, but it won't help anyone :-P I guess what people might really want is to take that JS and bundle it in an executable, like Deno does. It shouldn't bee to hard to do, I think. The problem is that, of course, one will want bundling to happen, so there should be a bundler too. I've been thinking about integrating esbuild by dynamically downloading the right binary when necessary, and using a subprocess for running the bundle step.

So that would give us tjs bundle which we can then use to implement some tjs compile which results in an exe...

[lal12]

Including a bundler would probably increase the size significantly. So it definitely should be excludable in build. Also there is the issue of needing more bundling features:

• Typescript
• React / JSX
• Some commonjs wrapping
• polyfills for NodeJS

This could get a lot quickly. Though maybe deno handles it in a clever way (never used it).

[KaruroChori]

Great, I will provide an update once I finish the first proof of concept so that we can review it.

Probably I misunderstood the intention of the issue I referenced. That being said... yes, an esbuild integration would be great. It would also allow tjs to seamlessly work with typescript by preprocessing the input stream and dropping types (excluding some introspection problems with the source file).
However it is written in go, which means garbage collection, which means there is no way I can use it :D. And the closest alternative would be swc written in rust, but it comes with the little issue of being 50MB!

[saghul]

@lal12 That's why I suggested esbuild. The idea would be not to embed it (it's written in Go so I'm not even sure how that would look like) but to download the (correct) binary to $TJS_HOME/bin/ and run it. The size increase should just be a bit of supporting JS code.

[lal12]

I think I like it, there would be some stuff I wanted to add for my build too. Though there are CMake ways to handle stuff like this. Not that I am the greatest fan of cmake but if it's used it probably should be used consistently.

[KaruroChori]

To be honest I generally use meson and not cmake of which I have limited experience; so, if you have any suggestion on how to make things more idiomatic that would be great!

[saghul]

@lal12 if we can narrow the scope even more with some CMake trickery, I'd be all for it!

[jspngh]

Hi @KaruroChori,

I wanted to join the discussion since you mentioned the example of lv_binding_js.
Personally I am also interested in having a better integration with txiki.js for this project.
A couple of days ago, I created a PR (#517) to allow using txiki.js as a library and have refactored lv_binding_js
to make use of this, which was pretty doable. I still have some work to do, but am planning on creating a PR
to propose these changes.

I think both for this repository and lv_binding_js, one of these two approaches (txiki.js as a library in the 'main' application
or extending txiki.js with external modules) should be chosen, since they are not really compatible.

I don't have a big preference or opinion, but intuitively for me, it makes more sense to have txiki.js as a dependency in a project that wants to extend it.
How would you compare your proposal compared with the "txiki.js as library" approach?

And I agree that it would be nice if some features in the core, like wasm, could be disabled.

[KaruroChori]

I think that both approaches are workable and they differ enough in scope and application. I would even say they could be used at the same time if needed, as they appear to be quite orthogonal.

My understanding is that your approach is about exposing artifacts, so that the parent application can link its native code to them directly. Which is great, but I fail to see how easily you would be able to integrate new custom functionality and expose it as part of the runtime since it also involves js2bytecode compilation at some stage.

What I was suggesting was a pre-build step in which code is added from a list of external modules, of which one or more can integrate custom functionality which is specific for your application. After that, the original build process of txiki continues mostly untouched.
So yes, for your application specific code, you either parent txiki or you attach your stuff as an external module; the two approaches are mutually exclusive, at least in practice.

But let's say you need some functionality for your application that you are not directly controlling and is not logically part of your basecode: for example bindings to handle the GPIO pins at high frequency on a raspberry pi, or, again, bindings for lvgl.
In that case I would say distributing them as an external package makes much more sense as you let the end developer decide how they want to compose their own distribution of txiki alongside other external modules and not just yours.

Then, as a developer, one can decide if their application specific code is better integrated as an external module, or by parenting it via cmake. But that is an end-of-the-chain choice in my opinion. For those providing bits of code which is supposed to be reusable, I think modules are the better approach.

[KaruroChori]

I still have some work to do, but am planning on creating a PR to propose these changes.

Please ping me once that is done! I would love to use it :D!

[jspngh]

Thanks for the explanation, I think I understand it better now.
It would be pretty cool to have plug-in modules that extend txiki.js,
and indeed the two approaches seem to be more orthogonal than I first assumed.

For LVGL bindings, I don't fully know how it would work as an external module.
Since you have to initialize LVGL and create a timer for rendering, I was assuming the bindings need
control over the main, but maybe that init logic could be exposed as a JS function that the user needs to call?

For adding new functionality to the runtime when using txiki.js as a library, I was thinking the same approach
can be used as for the core:

• using tjsc to create the byte code before compilation
• compile the byte code together with your application
• call tjs__eval_bytecode after you created a runtime with TJS_NewRuntime

But I haven't tried it yet, so no guarantee that this works without issues.

I still have some work to do, but am planning on creating a PR to propose these changes.

Please ping me once that is done! I would love to use it :D!

Will do!

[KaruroChori]

For reference, you can check the current progress on the demo here (not fully working yet):
https://github.com/KaruroChori/txiki.js/tree/module-manager
https://github.com/KaruroChori/txiki-module-demo
Now there is a proper PR #524

What you described seems to be more or less the way modules are supposed to work except for the last step.
If you have any feedback it will be welcome :).

[KaruroChori]

I dropped the idea of implementing the lvgl compatibility layer as a module after getting a half-working porting.
While possible, it would require to further extend modules with additional mechanisms, and no other module would be leveraging these additional features. To be honest, I cannot justify the effort of doing that when your approach is plenty good already :D.
So, for now no lvgl-txiki-module.

However I will probably wrap react as a module, so that I don't end up with two copies, one in the runtime and one bundled with the application running.

[KaruroChori]

@jspngh Based on your updated code, I also started porting the quickjs bindings for lvgl as an external module.

I opened https://github.com/KaruroChori/txiki-modules to track the selected packages I have been adapting/porting as txiki modules.

[jspngh]

So you found the branch in my lv_binding_js fork, right?
I just need to update the demo applications and then it should be more or less finished,
but I want to do some more testing with my actual application before I make a PR.

I'll be interested in how you tackle it in your approach, I'll keep an eye out.

[KaruroChori]

Yes, I also left some fixes as a PR in your fork to make it work on my system. I was able to get both demos working as expected.

[Tobbe]

I'm not so sure. I could easily add a tjs compile foo.js thing right now, but it won't help anyone :-P I guess what people might really want is to take that JS and bundle it in an executable, like Deno does.

Sorry for my lack of understanding here, but what would the most naive implementation of tjs compile do if not produce an executable?

It shouldn't bee to hard to do, I think. The problem is that, of course, one will want bundling to happen, so there should be a bundler too. I've been thinking about integrating esbuild by dynamically downloading the right binary when necessary, and using a subprocess for running the bundle step.

I'd be very interested in playing around with tjs compile even if it only supported a single file and no bundling! If/when needed I can handle the bundling into a single file myself first, before passing it on to tjs compile. I've been spending some time with Vite lately, so have a fairly good understanding of the bundling side of things.

[KaruroChori]

Sorry for my lack of understanding here, but what would the most naive implementation of tjs compile do if not produce an executable?

It is my understanding that the original (Bellard's) quickjs created an executable, but this was not portable and had assumptions over the toolchain used on the host system. The fork of quickjs used by txiki generates a C code containing the bytecode, which can be compiled for any platform.

But @saghul can surely provide a better answer to that.

[Tobbe]

was not portable

I've also found on of @saghul's other projects, https://github.com/saghul/njk, and that one is also not portable (doesn't work on Windows).

I hope 🤞 this won't be the case for binaries created by tjs compile (If any of you need access to a Windows computer, let me know and I'll see what I can do)