Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1.11.0-alpha2: precompilation fails if JULIA_DEPOT_PATH is set #53983

Closed
matthias314 opened this issue Apr 6, 2024 · 5 comments · Fixed by #54739 or #55066
Closed

1.11.0-alpha2: precompilation fails if JULIA_DEPOT_PATH is set #53983

matthias314 opened this issue Apr 6, 2024 · 5 comments · Fixed by #54739 or #55066
Labels
domain:packages Package management and loading
Milestone
1.11

Comments

@matthias314
Copy link
Contributor

matthias314 commented Apr 6, 2024 •
edited
Loading

This is a duplicate of #53831. I'm creating a new issue because I now have a reproducible MWE and because the old issue was apparently not noticed.

Setup: There is no ~/.julia, and JULIA_DEPOT_PATH is set to the empty directory /tmp/julia-depot. Then I try to install and use a package as follows:

~$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-alpha2 (2024-03-18)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> versioninfo()
Julia Version 1.11.0-alpha2
Commit 9dfd28ab751 (2024-03-18 20:35 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i3-10110U CPU @ 2.10GHz
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, skylake)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)
Environment:
  JULIA_DEPOT_PATH = /tmp/julia-depot

(@v1.11) pkg> st
  Installing known registries into `/tmp/julia-depot`
Status `/tmp/julia-depot/environments/v1.11/Project.toml` (empty project)

(@v1.11) pkg> add Chairmarks
    Updating registry at `/tmp/julia-depot/registries/General.toml`
   Resolving package versions...
   Installed Chairmarks ─ v1.2.1
    Updating `/tmp/julia-depot/environments/v1.11/Project.toml`
  [0ca39b1e] + Chairmarks v1.2.1
    Updating `/tmp/julia-depot/environments/v1.11/Manifest.toml`
  [0ca39b1e] + Chairmarks v1.2.1
  [de0858da] + Printf v1.11.0
  [4ec0a83e] + Unicode v1.11.0
Precompiling project...
  ✓ Unicode
  ✓ Printf
  3 dependencies successfully precompiled in 3 seconds
  2 dependencies precompiled but different versions are currently loaded. Restart julia to access the new versions

julia> using Chairmarks
[ Info: Precompiling Chairmarks [0ca39b1e-fe0b-4e98-acfc-b1656634c4de] (cache misses: wrong dep version loaded (2))
┌ Warning: Module Printf with build ID fafbfcfd-be51-2905-0028-424d602318e3 is missing from the cache.
│ This may mean Printf [de0858da-6303-5e67-8744-51eddeeeb8d7] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2319
┌ Info: Skipping precompilation due to precompilable error. Importing Chairmarks [0ca39b1e-fe0b-4e98-acfc-b1656634c4de].
└   exception = nothing

julia>

The check marks (✓) in front of Unicode and Printf are actually red and not green.

Restarting:

~$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-alpha2 (2024-03-18)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.11) pkg> st
Status `/tmp/julia-depot/environments/v1.11/Project.toml`
  [0ca39b1e] Chairmarks v1.2.1

julia> using Chairmarks
[ Info: Precompiling Chairmarks [0ca39b1e-fe0b-4e98-acfc-b1656634c4de] (cache misses: wrong dep version loaded (2))
┌ Warning: Module Printf with build ID fafbfcfd-be51-2905-0028-424d602318e3 is missing from the cache.
│ This may mean Printf [de0858da-6303-5e67-8744-51eddeeeb8d7] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2319
┌ Info: Skipping precompilation due to precompilable error. Importing Chairmarks [0ca39b1e-fe0b-4e98-acfc-b1656634c4de].
└   exception = nothing

julia>

If JULIA_DEPOT_PATH is not set, then the problem disappears.
@matthias314 matthias314 mentioned this issue Apr 6, 2024
Closed
@giordano giordano added the domain:packages Package management and loading label Apr 6, 2024
@giordano giordano added this to the 1.11 milestone Apr 6, 2024
@vtjnash
Copy link
Sponsor Member

vtjnash commented Apr 7, 2024

It is a problem with #53326, which I was aware of there but didn't implement the handling for this

@LilithHafner
Copy link
Member

@vtjnash, is this something you can take on?

@vtjnash
Copy link
Sponsor Member

vtjnash commented Jun 6, 2024

Probably not in the short term right now

vtjnash added a commit that referenced this issue Jun 8, 2024
@vtjnash
upgradable stdlibs
2c32529 
Fixes #53983

Unfortunately, TOML has a design problem with the way it is used by Pkg,
so this does not yet work correctly. Need to finish changing uses of
loaded_packages to explicit_loaded_modules in loading to try to make it
less terribly broken.
vtjnash added a commit that referenced this issue Jun 8, 2024
@vtjnash
upgradable stdlibs
ccfd971 
Fixes #53983

Unfortunately, TOML has a design problem with the way it is used by Pkg,
so this does not yet work correctly. Need to finish changing uses of
loaded_packages to explicit_loaded_modules in loading to try to make it
less terribly broken.

Also currently one test is broken, as we need some more code
rearrangement to make it legal again for an `__init__` method to import code.
@vtjnash vtjnash mentioned this issue Jun 8, 2024
Merged
vtjnash added a commit that referenced this issue Jun 20, 2024
@vtjnash
upgradable stdlibs
2b8201a 
Fixes #53983

This breaks one test, which relied on it being undefined behavior to
rely on the execution of `__init__` having occurred before `using` is
allowed to return. That usability issue is now (partially) eliminated.
vtjnash added a commit that referenced this issue Jun 22, 2024
@vtjnash
fix issue with loading stdlibs stalecheck inconsistency
2cf14aa 
This was an accidental late change in the PR, and I forgot why this
needed to be exactly written this way (and why this wasn't equivalent).

Followup to #54739 fixing #53983
@robsmith11
Copy link
Contributor

This seems to be broken still in 1.11.0-rc1 (which was released after this issue was resolved):

$ JULIA_DEPOT_PATH=/home/user/data/dot_julia /home/user/data/julia-1.11.0-rc1/bin/julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-rc1 (2024-06-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using DataFrames
[ Info: Precompiling DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0] (cache misses: wrong dep version loaded (2))
┌ Warning: Module REPL with build ID fafbfcfd-9cbb-b77f-0014-e2674d4b007c is missing from the cache.
│ This may mean REPL [3fa0cd96-eef1-5676-8a61-b3b8758bbffb] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2360
┌ Info: Skipping precompilation due to precompilable error. Importing DataFrames [a93c6f00-e57d-5684-b7b6-d8193f3e46c0].
└   exception = nothing
[ Info: Precompiling SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c] (cache misses: wrong dep version loaded (2))
┌ Warning: Module InteractiveUtils with build ID fafbfcfd-4d14-9bf7-0014-e2656d1ecfe3 is missing from the cache.
│ This may mean InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2360
┌ Info: Skipping precompilation due to precompilable error. Importing SortingAlgorithms [a2af1166-a08f-5f64-846c-94a0d3cef48c].
└   exception = nothing
[ Info: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8] (cache misses: wrong dep version loaded (2))
┌ Warning: Module InteractiveUtils with build ID fafbfcfd-4d14-9bf7-0014-e2656d1ecfe3 is missing from the cache.
│ This may mean InteractiveUtils [b77e0a4c-d291-57a0-90e8-8db25a27a240] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2360
┌ Info: Skipping precompilation due to precompilable error. Importing DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8].
└   exception = nothing
[ Info: Precompiling PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d] (cache misses: wrong dep version loaded (2))
┌ Warning: Module Markdown with build ID fafbfcfd-2939-cd23-0014-e264ef460a86 is missing from the cache.
│ This may mean Markdown [d6f4376e-aef5-505a-96c1-9c027394607a] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2360
┌ Info: Skipping precompilation due to precompilable error. Importing PrettyTables [08abe8d2-0d0c-5749-adfa-8a2ac140af0d].
└   exception = nothing

@matthias314
Copy link
Contributor Author

Just for reference: the issue mentioned in #54739 (comment) also hasn't been fully resolved in 11.0-rc1. With JULIA_DEPOT_PATH set and a startup.jl containing the line using InteractiveUtils I get:

~$ /tmp/julia-1.11.0-rc1/bin/julia
┌ Warning: Replacing module `Base64`
└ @ Base loading.jl:2263
┌ Warning: Replacing module `Markdown`
└ @ Base loading.jl:2263
┌ Warning: Replacing module `InteractiveUtils`
└ @ Base loading.jl:2263
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.0-rc1 (2024-06-25)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia>

@robsmith11 robsmith11 mentioned this issue Jul 5, 2024
Closed
@ViralBShah ViralBShah reopened this Jul 7, 2024
vtjnash added a commit that referenced this issue Jul 8, 2024
@vtjnash
fix loading of repeated/concurrent modules
a0c1326 
More followup to fix issues with require. There was an accidental
variable reuse (build_id) that caused it to be unable to load cache
files in many cases. There was also missing check for a dependency
already being loaded, resulting in trying to load it twice. Finally, the
start_loading code may drop the require_lock, but the surrounding code
was not prepared for that. Now integrate the necessary checks into
start_loading, instead of needing to duplicate them before and
afterwards.

Fixes #53983
Fixes #54940
Closes #55064
@vtjnash vtjnash mentioned this issue Jul 8, 2024
Merged
vtjnash added a commit that referenced this issue Jul 8, 2024
@vtjnash
fix loading of repeated/concurrent modules
9460d5c 
More followup to fix issues with require. There was an accidental
variable reuse (build_id) that caused it to be unable to load cache
files in many cases. There was also missing check for a dependency
already being loaded, resulting in trying to load it twice. Finally, the
start_loading code may drop the require_lock, but the surrounding code
was not prepared for that. Now integrate the necessary checks into
start_loading, instead of needing to duplicate them before and
afterwards.

Fixes #53983
Fixes #54940
Closes #55064
vtjnash added a commit that referenced this issue Jul 8, 2024
@vtjnash
fix loading of repeated/concurrent modules
7b70775 
More followup to fix issues with require. There was an accidental
variable reuse (build_id) that caused it to be unable to load cache
files in many cases. There was also missing check for a dependency
already being loaded, resulting in trying to load it twice. Finally, the
start_loading code may drop the require_lock, but the surrounding code
was not prepared for that. Now integrate the necessary checks into
start_loading, instead of needing to duplicate them before and
afterwards.

Fixes #53983
Fixes #54940
Closes #55064
KristofferC pushed a commit that referenced this issue Jul 23, 2024
@vtjnash @KristofferC
finish implementation of upgradable stdlibs (#54739)
19f838f 
This now allows the user to load any number of copies of a module, and
uses the combination of the environment, explicitly loaded modules, and
the requirements of the precompile caches to determine the meaning of a
name and which files need to be loaded. Note however that package
extensions continue to primarily only apply to the explicitly loaded
modules, although they may get loaded incidentally as the dependency of
another package, they won't get defined for every pair of combinations
of triggering modules.

Fixes #53983

(cherry picked from commit a1a2ac6)
KristofferC pushed a commit that referenced this issue Jul 23, 2024
@vtjnash @KristofferC
fix loading of repeated/concurrent modules (#55066)
d4f9808 
More followup to fix issues with require. There was an accidental
variable reuse (build_id) that caused it to be unable to load cache
files in many cases. There was also missing check for a dependency
already being loaded, resulting in trying to load it twice. Finally, the
start_loading code may drop the require_lock, but the surrounding code
was not prepared for that. Now integrate the necessary checks into
start_loading, instead of needing to duplicate them before and
afterwards.

Fixes #53983
Fixes #54940
Closes #55064

(cherry picked from commit fba928d)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
domain:packages Package management and loading
Projects
None yet
Milestone
1.11
6 participants
@vtjnash @ViralBShah @giordano @robsmith11 @matthias314 @LilithHafner