`resolve` too eager on local project #83

t-bltg · 2023-03-31T07:20:47Z

Reproducer (linux):

# in an empty directory
$ julia --project=. -e '
import Pkg; Pkg.add("CondaPkg")
import CondaPkg; CondaPkg.add("python")'
$ ls -a
.  ..  .CondaPkg  Manifest.toml  Project.toml
$ julia --project=. -e '
import Pkg; Pkg.status("CondaPkg")
import CondaPkg; CondaPkg.resolve()'  # noop (expected)
$ mkdir foo  # this triggers a `mtime` change
$ julia --project=. -e '
import Pkg; Pkg.status("CondaPkg")
import CondaPkg; CondaPkg.resolve()'  # `stat(dir).mtime` has changed, `_resolve_can_skip_1` fails, downloads conda-forge index (which on a low bandwidth network is expensive), etc ...

I don't want CondaPkg to fully resolve my pythonic dependencies anytime a file or folder is modified in a local julia project !

I'm using JULIA_CONDAPKG_ENV for a shared env and JULIA_CONDAPKG_BACKEND set to MicroMamba.

The text was updated successfully, but these errors were encountered:

cjdoris · 2023-04-24T07:44:33Z

That's not supposed to happen. Checking timestamps is just a quick way to avoid parsing dependencies. If the dependencies haven't actually changed, then CondaPkg shouldn't do anything.

Can you figure out what CondaPkg thinks has changed please?

t-bltg · 2023-04-24T17:29:43Z

This is the affected check:
https://github.com/cjdoris/CondaPkg.jl/blob/f9e6cc584c08edaeb5bbead2b076f54caf85677f/src/resolve.jl#L40

stat(dir).mtime changed because of the mkdir command in the reproducer, hence triggering full resolution.

cjdoris · 2023-05-14T15:04:01Z

Can you give me the full output from those commands please? Like I said, even if that check fails, CondaPkg shouldn't do anything if the dependencies haven't actually changed.

t-bltg · 2023-05-14T15:23:00Z

Here is the log

$ julia --project=. -e '
using Pkg; Pkg.add("CondaPkg")
using CondaPkg
CondaPkg.add("python")
'
   Resolving package versions...
    Updating `~/Downloads/foo/Project.toml`
  [992eb4ea] + CondaPkg v0.2.18
    Updating `~/Downloads/foo/Manifest.toml`
  [992eb4ea] + CondaPkg v0.2.18
  [692b3bcd] + JLLWrappers v1.4.1
  [0f8b85d8] + JSON3 v1.12.0
  [0b3b1443] + MicroMamba v0.1.12
  [69de0a69] + Parsers v2.5.8
  [fa939f87] + Pidfile v1.3.0
  [21216c6a] + Preferences v1.4.0
  [6c6a2e73] + Scratch v1.2.0
  [66db9d55] + SnoopPrecompile v1.0.3
  [856f2bd8] + StructTypes v1.10.0
  [f8abcde7] + micromamba_jll v1.4.1+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.9.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.2+0
  [14a3606d] + MozillaCACerts_jll v2022.10.11
  [83775a58] + Zlib_jll v1.2.13+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
    CondaPkg Found dependencies: $MAMBA_ROOT_PREFIX/CondaPkg.toml
    CondaPkg Found dependencies: [...]/.julia/packages/PythonCall/dsECZ/CondaPkg.toml
    CondaPkg Found dependencies: [...]/.julia/packages/PythonPlot/f591M/CondaPkg.toml
    CondaPkg Resolving changes
             + libstdcxx-ng
             + matplotlib
             + python
    CondaPkg Installing packages
             │ $JULIA_DEPOT_PATH/artifacts/a7fe6379eb776b1ec5b4ee1ddb8da76446d3f735/bin/micromamba
             │ -r $MAMBA_ROOT_PREFIX
             │ install
             │ -y
             │ -p $JULIA_CONDAPKG_ENV
             │ --override-channels
             │ --no-channel-priority
             │ libstdcxx-ng[version='>=3.4,<13.0',channel='conda-forge']
             │ matplotlib[version='>=1']
             │ python[version='*']
             │ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']
             └ -c conda-forge

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache

Transaction

  Prefix: $JULIA_CONDAPKG_ENV

  Updating specs:

   - conda-forge::libstdcxx-ng[version='>=3.4,<13.0']
   - matplotlib[version='>=1']
   - python=*
   - conda-forge::python[version='>=3.7,<4',build=*cpython*]


  Package         Version  Build               Channel           Size
───────────────────────────────────────────────────────────────────────
  Reinstall:
───────────────────────────────────────────────────────────────────────

  o libstdcxx-ng   12.2.0  h46fd767_19         conda-forge     Cached
  o python        3.10.11  he550d4f_0_cpython  conda-forge     Cached

  Summary:

  Reinstall: 2 packages

  Total download: 0 B

───────────────────────────────────────────────────────────────────────



Transaction starting
Changing libstdcxx-ng-12.2.0-h46fd767_19 ==> libstdcxx-ng-12.2.0-h46fd767_19
Changing python-3.10.11-he550d4f_0_cpython ==> python-3.10.11-he550d4f_0_cpython
Transaction finished

$ julia --project=. -e 'import CondaPkg; CondaPkg.resolve()'
$ mkdir foo
$ julia --project=. -e 'using CondaPkg; CondaPkg.resolve()'
  CondaPkg Found dependencies: $MAMBA_ROOT_PREFIX/CondaPkg.toml
  CondaPkg Found dependencies: [...]/.julia/packages/PythonCall/dsECZ/CondaPkg.toml
  CondaPkg Found dependencies: [...]/.julia/packages/PythonPlot/f591M/CondaPkg.toml
  CondaPkg Installing packages
           │ $JULIA_DEPOT_PATH/artifacts/a7fe6379eb776b1ec5b4ee1ddb8da76446d3f735/bin/micromamba
           │ -r $MAMBA_ROOT_PREFIX
           │ install
           │ -y
           │ -p $JULIA_CONDAPKG_ENV
           │ --override-channels
           │ --no-channel-priority
           │ libstdcxx-ng[version='>=3.4,<13.0',channel='conda-forge']
           │ matplotlib[version='>=1']
           │ python[version='*']
           │ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']
           └ -c conda-forge

                                         __
        __  ______ ___  ____ _____ ___  / /_  ____ _
       / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
      / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
     / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
    /_/

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache

Transaction

Prefix: $JULIA_CONDAPKG_ENV

Updating specs:

 - conda-forge::libstdcxx-ng[version='>=3.4,<13.0']
 - matplotlib[version='>=1']
 - python=*
 - conda-forge::python[version='>=3.7,<4',build=*cpython*]


Package         Version  Build               Channel           Size
───────────────────────────────────────────────────────────────────────
Reinstall:
───────────────────────────────────────────────────────────────────────

o libstdcxx-ng   12.2.0  h46fd767_19         conda-forge     Cached
o python        3.10.11  he550d4f_0_cpython  conda-forge     Cached

Summary:

Reinstall: 2 packages

Total download: 0 B

───────────────────────────────────────────────────────────────────────



Transaction starting
Changing libstdcxx-ng-12.2.0-h46fd767_19 ==> libstdcxx-ng-12.2.0-h46fd767_19
Changing python-3.10.11-he550d4f_0_cpython ==> python-3.10.11-he550d4f_0_cpython
Transaction finished

It's not the first time I see python and libstdcxx-ng being reinstalled without version change.

cjdoris · 2023-05-14T15:41:30Z

Ok, I assume you're using a shared Conda environment? e.g. you're setting JULIA_CONDAPKG_ENV?

In this case, the current logic is to reinstall all packages, because it's possible that something other than CondaPkg changed the environment. I guess this could be changed to not do this, and to assume that the user knows the pitfalls of shared environments.

t-bltg · 2023-05-14T15:42:39Z

Yes, this is what I specified in the original issue (#83 (comment)).

It would be convenient to have a way to skip reinstall, yes, especially since resolve wasn't called with the force argument.

cjdoris · 2023-05-17T17:10:27Z

Can you check #90 solves the issue please.

cjdoris · 2023-05-17T17:42:05Z

Huh. In that case could you work out what checks are causing resolve() to end up calling _resolve_conda_install() on line 509 or 531 of resolve.jl please?

t-bltg · 2023-05-18T12:25:06Z

@cjdoris, please ignore my previous comment, I was wrong because I dev'ed my patched Condapkg with #90 applied in my default env and not in the local project.

I can confirm that #90 solves the most annoying part of specific issue (no reinstallation nor conda index download, even if the dependencies (.toml) are still searched for (this is IMO minor, no significative perf penalty)).

cjdoris · 2023-05-18T19:10:14Z

Ok awesome!

t-bltg mentioned this issue Mar 31, 2023

simplifications #82

Merged

cjdoris mentioned this issue May 17, 2023

do not reinstall all packages just because the env is shared #90

Merged

This comment was marked as outdated.

Sign in to view

cjdoris closed this as completed in #90 Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`resolve` too eager on local project #83

`resolve` too eager on local project #83

t-bltg commented Mar 31, 2023 •

edited

Loading

cjdoris commented Apr 24, 2023

t-bltg commented Apr 24, 2023

cjdoris commented May 14, 2023

t-bltg commented May 14, 2023

cjdoris commented May 14, 2023

t-bltg commented May 14, 2023 •

edited

Loading

cjdoris commented May 17, 2023

This comment was marked as outdated.

cjdoris commented May 17, 2023

This comment was marked as outdated.

t-bltg commented May 18, 2023 •

edited

Loading

cjdoris commented May 18, 2023

resolve too eager on local project #83

resolve too eager on local project #83

Comments

t-bltg commented Mar 31, 2023 • edited Loading

cjdoris commented Apr 24, 2023

t-bltg commented Apr 24, 2023

cjdoris commented May 14, 2023

t-bltg commented May 14, 2023

cjdoris commented May 14, 2023

t-bltg commented May 14, 2023 • edited Loading

cjdoris commented May 17, 2023

This comment was marked as outdated.

cjdoris commented May 17, 2023

This comment was marked as outdated.

t-bltg commented May 18, 2023 • edited Loading

cjdoris commented May 18, 2023

`resolve` too eager on local project #83

`resolve` too eager on local project #83

t-bltg commented Mar 31, 2023 •

edited

Loading

t-bltg commented May 14, 2023 •

edited

Loading

t-bltg commented May 18, 2023 •

edited

Loading