Rework che-theia build process to not use source linking

[tsmaeder]

Currently, we're linking source folders in order build che theia, we have the following structure ('->' being a symlink):

/theia
  /.git
  ...
  /plugins
    ->/theia/che/che-theia/plugins/task-plugin
    ...
  /extensions
    ->/theia/che/che-theia/extensions/eclipse-che-theia-about
  /che
    /che-theia
    /.git
    ...
    /plugins
      /task-plugin
    /extensions
      /eclipse-che-theia-about

The problem with this is that the build process does not handle source links very well: for one things, there will be two node_modules folder: one inside /theia and one inside /theia/che/che-theia. The fact that cd /projects/theia && yarn becomes broken when you rebuild a che plugin or extension seems to be a consequence of this (deleting both node_modules directories fixes the problem).
Since the build only works if the che-theia folder is located inside the theia folder, there is an additional drawback: the theia git tools cannot be used for changes in che-theia, because theia can't handle the case of nested git repos, it seems.

My proposal is to move to a setup that is based on yarn link. The idea is to have che-theia as a separate repository and to yarn link $package the theia packages in the appropriate places. The setup would look like this:

/theia
...just a vanilla theia install with all the packages linked to ~/.config/yarn/link
/che-theia
/assembly
...
/extensions
...
/plugins
...

I was able to almost make this work in a day an it seemed to address the problems mentioned above. I think investing some more time would be worth the time.

[benoitf]

I think it's something that can be done properly/smoothly only with yarn 2

[tsmaeder]

Let's give it a try, shall we?

[tsmaeder]

And why would it not work with yarn 1.x?

[tsmaeder]

@benoitf ?

[benoitf]

One issue is yarn link is considering links to be folders, not packages
so there are issues with transitive dependencies / peer dependencies
yarnpkg/yarn#2914

portal: might be the savior:
https://dev.to/arcanis/introducing-yarn-2-4eh1#new-protocol-raw-portal-endraw-

https://yarnpkg.com/features/protocols#whats-the-difference-between-link-and-portal

Also the fact that we can store the dependencies should improve velocity.

[tsmaeder]

@benoitf I've read the linked issue, but I don't really understand why it would block us: the issues seems to state the yarn does not call install on the library packages (in our case: the theia packages). But we could do that ourselves when necessary.
What we need is this:

1. For our packages to be compiled against the correct version of upstream packages, assuming we have built those correct.y
2. When packaging a build target with webpack or other tools, we need to pick up all dependencies and the proper version of the dependencies

Could you spell out how the yarn link behavior is breaking those requirements? It is not clear to me from reading the issue.

[tsmaeder]

IMO, two further changes could be made to make the build even simpler:

1. Don't have che-theia init override the theia version in the package.json files: it makes it impossible to check in what was tested. Instead, check in the version that is used.
2. Don't generate the assembly in che-theia init. Instead, check it into the repo

[nickboldt]

No immediate concerns w/ this for downstream, assuming that the existing method for getting che-theia code into crw-theia remains compatible after your changes (or else we'd rework the crw build.sh script to accommodate your changes).

[azatsarynnyy]

It also should be checked that the proposed changes don't break the release flow.
The release script relies on some important files, such as:

• version field of all the package.json files
• build.include
• che-theia-init-sources.yml

[sunix]

Build done in CI and developer environment (che-theia in che) should be using the same script.

[tsmaeder]

I believe the concrete steps to implement linkless builds would be the following:

1. move the static assembly inside the che-theia project and commit it with the repo and remove the assembly generation step from che-theia init. I think the code is static enough that we can just run it when needed. When that is needs to be analyzed.
2. Add a "--theia-version=" parameter to che-theia init. All dependencies on @Theia packages in che-theia will be changed to this version. Default is "next". So by default, no changes will be made to the checked-in code
3. Add a "--theia-source=" parameter. If present, che-theia init will clone theia and checkout the given version. The appropriate packages are yarn-linked in the che-theia/node_modules folder. We should support checking out branches, tags and specific commits as well as forks of theia. If the parameter is missing, we don't check anything out but build against @Theia packages from the npm registry.
4. fix build scripts to use the new parameters.
5. fix build scripts to package theia and che-theia artifacts from the new correct locations

[tsmaeder]

Just found another case where our current structure is not working: After building plugins (.foreach_yarn), you'll have to clean out theia/che/che-theia/node_modules if you want to be sure that all extensions are finding the same node modules when you run the back end. There is a module called async-mutex that is only referenced by the che workspace plugins. As such, it will not be installed when you do a yarn in the theia folder and you have to install it manually.

[tsmaeder]

The plan of attack in #16852 (comment) has been changed slightly:

1. Provide che-theia commands for generate (generates assembly folder inside che-theia folder), link (sets up yarn linking to compile agains theia source) and 'updateDependencies` (updates the version of the relevant @Theia dependencies in all che-theia package.json files)
2. Fix build scripts to use the new targets instead of che-theia init
3. Remove che-theia init and related code.
4. Remove assembly generation step if possible.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.