Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encapsulation of polyglot/java Dependencies #7082

Open
1 task
radeusgd opened this issue Jun 20, 2023 · 15 comments · May be fixed by #10714
Open
1 task

Encapsulation of polyglot/java Dependencies #7082

radeusgd opened this issue Jun 20, 2023 · 15 comments · May be fixed by #10714
Assignees
Labels
-compiler x-new-feature Type: new feature request

Comments

@radeusgd
Copy link
Member

radeusgd commented Jun 20, 2023

With the imports/exports discussion we are talking about encapsulation of out private modules.

Many of our libraries rely on Java helpers that are imported using the polyglot java import statements. To achieve proper encapsulation of library internals, we should ensure that these internal helper libraries (std-base, std-table etc.) are private and not accessible from user projects.

High level idea

For most cases, we want to ensure that a JAR file that is in a polyglot directory of a given project can only be imported from within that project and not from other places.

Re-exporting

We still want some helpers to be visible in other modules. For example, some helpers from std-table (the Table library companion JAR), are also used in the Database library. To do so we can re-export such symbols by wrapping them in Enso wrappers. This will be enough for helper methods, but it may not be enough for types.

For the time being this should be enough for our usecases.
Potentially, for longer-term, maybe we should devise a better control over where the JARs are importable from. Maybe we can include some information in the package.yaml file or have a sibling directory public-polyglot - where such JARs would be visible to the outside world?

Implementation

Currently, loading a JAR is achieved by TruffleLanguage::Env::addToHostClassPath. This does not have the control for imports we need. We would probably need a check at compiler level which verifies if a given polyglot java import clause is allowed based on the contents of the polyglot directory associated with the project from which the compiled Enso file comes from.

This may require some non-trivial logic to scan each JAR added to the classpath and for each path it adds to the classpath, we would need to associate it with the project it comes from. Then the compiler can verify for each polyglot java import if the current project has the 'permission' to access a given path.

Collisions

This still leaves the question of - what if two libraries contain JARs in their polyglot directories that have clashing classpaths? For example, two versions of the same Java library? Ideally, due to the encapsulation each library should be able to import each own version, but that would require some stricter separation of classpaths.

Until we have some 'clever' solution for this, we may want to forbid importing clashing libraries. Ideally we should detect if adding a new library to the classpath will shadow another one and disallow importing such a library.

Alternatives

The Java Module System seems to be solving a similar task. Maybe we can somehow plug into it and exploit it instead of inventing our own wheels? It partially depends on if that is possible to do in how Truffle/Graal handles the modules for the polyglot interface.

Tasks

@JaroslavTulach
Copy link
Member

Isolation can be achieved by using different ClassLoader for each library. E.g. having different HostClassLoader for each library.

@JaroslavTulach JaroslavTulach changed the title Encapsulation of Polyglot Dependencies Encapsulation of polyglot/java Dependencies Jun 4, 2024
@JaroslavTulach JaroslavTulach linked a pull request Jul 30, 2024 that will close this issue
5 tasks
@JaroslavTulach JaroslavTulach linked a pull request Jul 30, 2024 that will close this issue
5 tasks
@JaroslavTulach
Copy link
Member

The implementation of encapsulation is provided by #10714. It uses Java Platform Module System and defines a ModuleLayer for each library with JARs in polyglot/java directory.

@enso-bot
Copy link

enso-bot bot commented Jul 31, 2024

Jaroslav Tulach reports a new STANDUP for yesterday (2024-07-30):

Progress: - #7082: ClassPath & Using ModuleLayer for loading classes: #10714

@enso-bot
Copy link

enso-bot bot commented Jul 31, 2024

Jaroslav Tulach reports a new STANDUP for today (2024-07-31):

Progress: - Turning libraries into modules by Pavel: 7e865ce#r1697852981

  • uses/provides: c4c1842#r1697927708
  • removal of add_to_class_path
  • turning std-table into Java module
  • pair IGV with Pavel
  • meeting James
  • introducing Base_Internal_Tests: 74fa9e1
  • standups & meetings It should be finished by 2024-08-05.

@enso-bot
Copy link

enso-bot bot commented Aug 2, 2024

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-01):

Progress: - libs as Java Modules

GitHub
Pull Request Description Implements #7082 by creating a module layer per library. Removes the obsolete add_to_class_path concept. Important Notes

Checklist
Please ensure that the following checkli...

@enso-bot
Copy link

enso-bot bot commented Aug 3, 2024

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-02):

Progress: - case study ModuleLayer project: https://github.com/JaroslavTulach/multilayerstudy

GitHub
Case study of using modules and module layers. Contribute to JaroslavTulach/multilayerstudy development by creating an account on GitHub.
GitHub
Case study of using modules and module layers. Contribute to JaroslavTulach/multilayerstudy development by creating an account on GitHub.

@enso-bot
Copy link

enso-bot bot commented Aug 4, 2024

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-03):

Progress: - WarningBenchmarks: #10555 (comment)

@JaroslavTulach
Copy link
Member

@gunnarmorling provided a reference to layrry - A Launcher and API for Modularized Java Applications.

@enso-bot
Copy link

enso-bot bot commented Aug 6, 2024

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-05):

Progress: - Deep review of #10725 (review)

GitHub
Case study of using modules and module layers. Contribute to JaroslavTulach/multilayerstudy development by creating an account on GitHub.
X (formerly Twitter)

@enso-bot
Copy link

enso-bot bot commented Aug 7, 2024

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-06):

Progress: - REPL script verified: 78db664

@enso-bot
Copy link

enso-bot bot commented Aug 8, 2024

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-07):

Progress: - #10736: error&warnings PR cleanup: #10765 (comment)

@enso-bot
Copy link

enso-bot bot commented Aug 9, 2024

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-08):

Progress: - Reimplemented: #10765 (comment)

GitHub
Enso is dynamically compiled language built on top of JVM - e.g. GraalVM - and its Truffle framework as such programs written in Enso can run pretty fast - once the dynamic compiler does its job. T...

@enso-bot
Copy link

enso-bot bot commented Aug 10, 2024

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-09):

Progress: - review of revert: #10778

@enso-bot
Copy link

enso-bot bot commented Aug 13, 2024

Jaroslav Tulach reports a new STANDUP for yesterday (2024-08-12):

Progress: - review: #10793 (review)

@JaroslavTulach
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
-compiler x-new-feature Type: new feature request
Projects
Status: 🔧 Implementation
Development

Successfully merging a pull request may close this issue.

4 participants