-
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Steel as an optional plugin system #8675
base: master
Are you sure you want to change the base?
Conversation
Is it possible to modify the behavior of |
@godalming123 did you try marksman lsp ? |
@noor-tg right now I'm using markdown-oxide, does the feature described work with marksman? |
@godalming123 I don't know. |
Hi @mattwparas, just a quick reminder not to stress about this PR. Not sure how you're handling the attention and millions of people worldwide at the edge of their seats, so I am going on a limb here and just remind you to take your time while we're enjoying the show. |
People shouldn't pressure him into finishing this as soon as possible, but one misconception that slows people down is the notion of relaxing. If you are going to do anything, try 100%, or don't do it. Taking it easy won't get things done. Positive stress is good. |
@amano-kenji this ain't LinkedIn |
I get the impression that big picture feedback is the helpful thing at this stage, but in order to come up with that feedback I feel like I need to overcome many tiny problems and use it for a while. The tiny problem that brought me here:
Is there a better place (besides this comments section) for discussing such things? |
I am not so convinced of this proposal. I share my thoughts, so you can think if this is helpful. I don't claim to be right. Scheme is a functional language. If you decide for it, you should embrace the functional style. And I think, Funcional Programming is indeed a good idea for Helix. I try to illustrate what I mean. In your proposal you pass around the cx. That looks to me a lot like Object Oriented Programming. Compare it for example with Python, where you have to pass around the self object. The fact that you only stir in cx does not make the design stateless. I have nothing against this style. This is how Emacs works. Plugins in Emacs operate on everything. That makes everything quite easy, but gets more complicated, when things need more coordination at runtime or at interface level. Emacs for example has no multi threading. At least it does not profit from it. For that way of working, I think Janet (https://janet-lang.org/) is a bit less functional and has more batteries included than Scheme and is perhaps a bit more mainstream oriented. It works and many people are happy with it. So I think, there is nothing against it. But I want to explain, why a proper functional approach in my opinion still fits better. In your example "Create a file under wherever we are" the plugin code is in full control of everything. It talks directly with the Helix core, requests data, and updates data. But imagine what happens if something there changes. For example you decide to introduce a graphical user interface and now the current mode is no longer FILE-TREE, but maybe NICE-UI-FILE-TREE or your selection is no longer a line number, because you have a GUI with real dialog buttons for that. That is hard to imagine. Why would someone want to develop an alternative UI for the FILE-TREE? But why not? How can you enable him to make his code compatible with your plugin? In Java you have to define tons of classes and interfaces. But in dynamic languages you don't have to. You can pass functions around or register functions somewhere. Only the name and the signature matter. So when developing an API, I think, you should not just expose all these Rust functions like The other point, that I see critical is that the plugin is in charge of everything. In Functional Programming you normally try to have the side effects at the boundaries. In Object Oriented Programming you may have heard of "Hexagonal Architecture" or "Ports and Adapters". That means, that you have one business logic that is in control of everything at a very abstract level. In your plugin, this would mean, you have one function Why is this better? Because you automatically establish interfaces for everything without thinking about them. In Java (and maybe in Rust?) such an architecture is hard to achieve. You have to read thick books to understand how to do it. In Functional Progamming you automatically work like this. Functional Programming automatically brings you on that track. You don't have to train people to do it the right way. There is no choice if you really enforce side effects at the boundaries. The result is that it is very easy to replace one functionality by another functionality without breaking hundreds of plugins. If the NICE-UI-FILE-TREE has a function It sounds complicated. But it is not. You only have to follow the idea that everything, which is side-effecty happens outside. You already have the lambdas. Return them instead of executing them. And a final though, why I think, Functional Programming fits better: You want Helix be resonsive at any time. Imagine you have 1000 cursors in a file. You write a plugin that does something for each of the cursors. For example fetching something from a database. For each location it takes 0.1s. Helix would be blocked for 100s. That is not acceptable. How can you do it better? We should not let the plugin in the control of this. We should only ask the plugin how to do things. The plugin delivers 1000 functions to the business logic. And the business logic decides when and how to execute them. If for example none of the 1000 working areas overlap, all 1000 functions can be executed in parallel. We can also give them a priority. For example if the user scrolls to a place, where any of the 1000 functions is pending, it can quickly be executed, so that the user can continue working. Or parts of the file that have pending changes can be made read only. You can provide a progress bar. You could even allow speculative editing. You cannot expect all this from a plugin developer. This must come from the business logic. And you only have to think once about these mechanisms. Then they are not complicated. What is the abstract model of this? It is quite simple: You have 1000 selections in the file and 1000 functions that do some local text manipulations on these selections. For plugin developers that would mean, they have to decide whether they do local modifications and follow that model or not. If they do non-local modifications, they will block Helix. |
A good choice may be Helix exposing APIs for implementing optional scripting languages and everyone can embed whatever they want.
Scheme due to its simplicity can be an igniter for such a plan.
Just some preliminary thoughts.
…________________________________
From: habruening ***@***.***>
Sent: Sunday, September 22, 2024 4:07:05 PM
To: helix-editor/helix ***@***.***>
Cc: Chuyang Chen ***@***.***>; Manual ***@***.***>
Subject: Re: [helix-editor/helix] Add Steel as an optional plugin system (PR #8675)
I am not so convinced of this proposal. I share my thoughts, so you can think if this is helpful. I don't claim to be right.
Scheme is a functional language. If you decide for it, you should embrace the functional style. And I think, Funcional Programming is indeed a good idea for Helix. I try to illustrate what I mean.
In your proposal you pass around the cx. That looks to me a lot like Object Oriented Programming. Compare it for example with Python, where you have to pass around the self object. The fact that you only stir in cx does not make the design stateless. I have nothing against this style. This is how Emacs works. Plugins in Emacs operate on everything. That makes everything quite easy, but gets more complicated, when things need more coordination at runtime or at interface level. Emacs for example has no multi threading. At least it does not profit from it.
For that way of working, I think Janet (https://janet-lang.org/) is a bit less functional and has more batteries included than Scheme and is perhaps a bit more mainstream oriented. It works and many people are happy with it. So I think, there is nothing against it.
But I want to explain, why a proper functional approach in my opinion still fits better.
In your example "Create a file under wherever we are" the plugin code is in full control of everything. It talks directly with the Helix core, requests data, and updates data. But imagine what happens if something there changes. For example you decide to introduce a graphical user interface and now the current mode is no longer FILE-TREE, but maybe NICE-UI-FILE-TREE or your selection is no longer a line number, because you have a GUI with real dialog buttons for that. That is hard to imagine. Why would someone want to develop an alternative UI for the FILE-TREE? But why not? How can you enable him to make his code compatible with your plugin?
In Java you have to define tons of classes and interfaces. But in dynamic languages you don't have to. You can pass functions around or register functions somewhere. Only the name and the signature matter. So when developing an API, I think, you should not just expose all these Rust functions like currently-in-labelled-buffer? or temporarily-switch-focus. That is not an API. That is only another part of your program, which is compiled differently. You should make things abstract and give them abstract names. But luckily it just happens when you do Functional Programming. You don't have to plan thousand aspects of you API upfront. Just start naming things in abstract ways and register them somewhere.
The other point, that I see critical is that the plugin is in charge of everything. In Functional Programming you normally try to have the side effects at the boundaries. In Object Oriented Programming you may have heard of "Hexagonal Architecture" or "Ports and Adapters". That means, that you have one business logic that is in control of everything at a very abstract level. In your plugin, this would mean, you have one function dirname-prompter that knows how to ask for a directory. But it does not actually executes this. It returns a function to the business logic, and the business logic can call it, when it wants. And then you have another function dir-creator that knows how to create a directory. So the business logic is: Asking for a directory and creating it. That is what the business logic has to implement. And that is a very abstract connection between two abstract things. It can also come from a plugin. So the control flow is: 1. The business logic determines what dirname-prompter needs. For example it must know how to use helix-prompt!. So the business logic creates a function around helix-prompt! and passes that function to dirname-prompter. Then dirname-prompter executes everything and delivers back something. The business logic does not even understand, what it gets. But it passes it to dir-creator together with everything else, that is needed by it.
Why is this better? Because you automatically establish interfaces for everything without thinking about them. In Java (and maybe in Rust?) such an architecture is hard to achieve. You have to read thick books to understand how to do it. In Functional Progamming you automatically work like this. Functional Programming automatically brings you on that track. You don't have to train people to do it the right way. There is no choice if you really enforce side effects at the boundaries.
The result is that it is very easy to replace one functionality by another functionality without breaking hundreds of plugins. If the NICE-UI-FILE-TREE has a function tell-me-a-new-dirname it will work, otherwise not. There are no surprises. But of course it only would work if the business logic provides it with the functionalities it needs. For read only file system it may not be possible. And for remote SSH connections it may not known how to create folders, because the plugin developer of the SSH connection did not provide the low level functions for creating folders. So the business logic would know that in these situations the plugin cannot be used.
It sounds complicated. But it is not. You only have to follow the idea that everything, which is side-effecty happens outside. You already have the lambdas. Return them instead of executing them.
And a final though, why I think, Functional Programming fits better: You want Helix be resonsive at any time. Imagine you have 1000 cursors in a file. You write a plugin that does something for each of the cursors. For example fetching something from a database. For each location it takes 0.1s. Helix would be blocked for 100s. That is not acceptable. How can you do it better? We should not let the plugin in the control of this. We should only ask the plugin how to do things. The plugin delivers 1000 functions to the business logic. And the business logic decides when and how to execute them. If for example none of the 1000 working areas overlap, all 1000 functions can be executed in parallel. We can also give them a priority. For example if the user scrolls to a place, where any of the 1000 functions is pending, it can quickly be executed, so that the user can continue working. Or parts of the file that have pending changes can be made read only. You can provide a progress bar. You could even allow speculative editing. You cannot expect all this from a plugin developer. This must come from the business logic. And you only have to think once about these mechanisms. Then they are not complicated. What is the abstract model of this? It is quite simple: You have 1000 selections in the file and 1000 functions that do some local text manipulations on these selections. For plugin developers that would mean, they have to decide whether they do local modifications and follow that model or not. If they do non-local modifications, they will block Helix.
—
Reply to this email directly, view it on GitHub<#8675 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFOXEBDEA66JQTOLI23N5PLZX4POTAVCNFSM6AAAAAA6XBKXQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRWHE2DGMZYHE>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
I don't think the maintainers want to support other scripting languages, but from what I understand it is designed with the possibility in mind in case we choose an inhouse scheme dialect. |
It might be better to just open an issue on my fork, that way I can help go through things without notifying everyone else. @habruening - I appreciate the commentary, however there are a few points that I'd like to call out.
For all of your ideas, I welcome an implementation. I don't think this is the direction that I will be taking this PR. @atahrijouti I appreciate the kind words. I am still working on this, just less publicly for the last few months. |
I also came prepared to bleed, so I'm not really after support so much as a public place to collect workarounds and document the sharp edges, even if fixing them ends up being low/no priority. Issues will do nicely. Can I trouble you to enable issues in that repo? 😀 |
@habruening The best UI paradigm I know is reactive values pioneered by https://github.com/keera-studios/keera-hails Reactive values are like spreadsheet cells that react to the values of other cells. The reactive values can be unidirectional or bidirectional. Functional reactive programming is best used for animation. Since helix editor is a user interface, reactive values are useful. |
I'd love to start playing around with this, but currently the flake.nix hasn't been updated to add the required files. |
Will steel scheme be as integrated as emacs lisp? Neovim just tacked lua on top of vimscript, and lua scripting in neovim is painful because lua is not very well integrated with neovim, and documentation is not very discoverable. Although emacs lisp lacks namespace and threads, it feels better than vimscript and lua. |
AFAIK Elisp is not used as some plugin system in Emacs. Emacs is written in Elisp! So the situation is very different and by definition Steel cannot be as tightly integrated in Helix as Elisp is in Emacs. |
At least, steel scheme is going to be better than lua because lua was added on top of vimscript. Working with vimscript legacy in lua was bad. |
@Talia-12 I have been working on this flake.nix. The devshell that it provides has I haven't tried to make these changes in the |
Optionally load init.scm and helix.scm from .helix/
A repo being able to run untrusted scm on load seems a bit scary. Maybe there could be a trusted workspace system for something like that similar to VSCode/Jetbrains IDEs? |
Notes:
Opening this just to track progress on the effort and gather some feedback. There is still work to be done but I would like to gather some opinions on the direction before I continue more.
You can see my currently functioning helix config here and there are instructions listed in the
STEEL.md
file. The main repo for steel lives here, however much documentation is in works and will be added soon.The bulk of the implementation lies in the
engine.rs
andscheme.rs
files.Design
Given prior conversation about developing a custom language implementation, I attempted to make the integration with Steel as agnostic of the engine as possible to keep that door open.
The interface I ended up with (which is subject to change and would love feedback on) is the following:
If you can implement this, the engine should be able to be embedded within Helix. On top of that, I believe what I have allows the coexistence of multiple scripting engines, with a built in priority for resolving commands / configurations / etc.
As a result, Steel here is entirely optional and also remains completely backwards compatible with the existing toml configuration. Steel is just another layer on the existing configuration chain, and as such will be applied last. This applies to both the
config.toml
and thelanguages.toml
. Keybindings can be defined via Steel as well, and these can be buffer specific, language specific, or global. Themes can also be defined from Steel code and enabled, although this is not as rigorously tested and is a relatively recent addition. Otherwise, I have been using this as my daily driver to develop for the last few months.I opted for a two tiered approach, centered around a handful of design ideas that I'd like feedback on:
The first, there is a
init.scm
and ahelix.scm
file - thehelix.scm
module is where you define any commands that you would like to use at all. Any function exposed via that module is eligible to be used as a typed command or via a keybinding. For example:This would then make the command
:shell
available, and it will just replace the%
with the current file. The documentation listed in the@doc
doc comment will also pop up explaining what the command does:Once the
helix.scm
module isrequire
'd - then theinit.scm
file is run. One thing to note is that thehelix.scm
module does not have direct access to a running helix context. It must act entirely stateless of anything related to the helix context object. Runninginit.scm
gives access to a helix object, currently defined as*helix.cx*
. This is something I'm not sure I particularly love, as it makes async function calls a bit odd - I think it might make more sense to make the helix context just a global inside of a module. This would also save the hassle that every function exposed has to accept acx
parameter - this ends up with a great deal of boilerplate that I don't love. Consider the following:Every function call to helix built ins requires passing in the
cx
object - I think just having them be able to reference the global behind the scenes would make this a bit ergonomic. The integration with the helix runtime would make sure whether that variable actually points to a legal context, since we pass this in via reference, so it is only alive for the duration of the call to the engine.Async functions
Steel has support for async functions, and has successfully been integrated with the tokio runtime used within helix, however it requires constructing manually the callback function yourself, rather than elegantly being able to use something like
await
. More to come on this, since the eventual design will depend on the decision to use a local context variable vs a global one.Built in functions
The basic built in functions are first all of the function that are typed and static - i.e. everything here:
However, these functions don't return values so aren't particularly useful for anything but their side effects to the editor state. As a result, I've taken the liberty of defining functions as I've needed/wanted them. Some care will need to be decided what those functions actually exposed are.
Examples
Here are some examples of plugins that I have developed using Steel:
File tree
Source can be found here
filetree.webm
Recent file picker
Source can be found here
recent-files.webm
This persists your recent files between sessions.
Scheme indent
Since steel is a scheme, there is a relatively okay scheme indent mode that only applied on
.scm
files, which can be found here. The implementation requires a little love, but worked enough for me to use helix to write scheme code 😄Terminal emulator
I did manage to whip up a terminal emulator, however paused the development of it while focusing on other things. When I get it back into working shape, I will post a video of it here. I am not sure what the status is with respect to a built in terminal emulator, but the one I got working did not attempt to do complete emulation, but rather just maintained a shell to interact with non-interactively (e.g. don't try to launch helix in it, you'll have a bad time 😄 )
Steel as a choice for a language
I understand that there is skepticism around something like Steel, however I have been working diligently on improving it. My current projects include shoring up the documentation, and working on an LSP for it to make development easier - but I will do that in parallel with maintaining this PR. If Steel is not chosen and a different language is picked, in theory the API I've exposed should do the trick at least with matching the implementation behavior that I've outlined here.
Pure rust plugins
As part of this, I spent some time trying to expose a C ABI from helix to do rust to rust plugins directly in helix without a scripting engine, with little success. Steel supports loading dylibs over a stable abi (will link to documentation once I've written it). I used this to develop the proof of concept terminal emulator. So, you might not be a huge fan of scheme code, but in theory you can write mostly Rust and use Steel as glue if you'd like - you would just be limited to the abi compatible types.
System compatibility
I develop off of Linux and Mac - but have not tested on windows. I have access to a windows system, and will get around to testing on that when the time comes.