esmvaltool-suite v2.5.0 #5

regro-cf-autotick-bot · 2022-03-15T18:17:46Z

It is very likely that the current package version for this feedstock is out of date.

Checklist before merging this PR:

Dependencies have been updated if changed: see upstream
Tests have passed
Updated license if changed and license_file is packaged

Information about this PR:

Feel free to push to the bot's branch to update this PR if needed.
The bot will almost always only open one PR per version.
The bot will stop issuing PRs if more than 3 version bump PRs generated by the bot are open. If you don't want to package a particular version please close the PR.
If you want these PRs to be merged automatically, make an issue with @conda-forge-admin,please add bot automerge in the title and merge the resulting PR. This command will add our bot automerge feature to your feedstock.
If this PR was opened in error or needs to be updated please add the bot-rerun label to this PR. The bot will close this PR and schedule another one. If you do not have permissions to add this label, you can use the phrase @conda-forge-admin, please rerun bot in a PR comment to have the conda-forge-admin add it for you.

Pending Dependency Version Updates

Here is a list of all the pending dependency version updates for this repo. Please double check all dependencies before merging.

Name	Upstream Version	Current Version
esmvaltool-suite	2.5.0

Dependency Analysis

Please note that this analysis is highly experimental. The aim here is to make maintenance easier by inspecting the package's dependencies. Importantly this analysis does not support optional dependencies, please double check those before making changes. If you do not want hinting of this kind ever please add bot: inspection: false to your conda-forge.yml. If you encounter issues with this feature please ping the bot team conda-forge/bot.

Analysis by source code inspection shows a discrepancy between it and the the package's stated requirements in the meta.yaml.

Packages found by source code inspection but not in the meta.yaml:

progressbar
prov
iris
pyamg
python-dateutil
fire
requests
iris-ued

Packages found in the meta.yaml but not found by source code inspection:

cython
scikit-image
rasterio
nc-time-axis
openpyxl
esmvaltool-ncl
esmvaltool-r

_{This PR was created by the regro-cf-autotick-bot. The regro-cf-autotick-bot is a service to automatically track the dependency graph, migrate packages, and propose package version updates for conda-forge. Feel free to drop us a line if there are any issues! This PR was generated by https://github.com/regro/autotick-bot/actions/runs/1988351915, please use this URL for debugging.}

…nda-forge-pinning 2022.03.15.10.12.29

conda-forge-linter · 2022-03-15T18:17:49Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

schlunma · 2022-03-16T08:45:49Z

Packages found by source code inspection but not in the meta.yaml:
* progressbar

* prov

* iris

* pyamg

* python-dateutil

* fire

* requests

* iris-ued

These are not present in out environment.yml; some of them are present in our setup.py. Should I add them here?

valeriupredoi · 2022-03-16T08:51:43Z

cheers Manu! We need to add direct dependencies, and not the ones that are already dependencies of our direct dependencies (and you can kick me if I say dependency one more time 😆 ) - so yeah, if it's in setup.py then that needs to be in meta (with the pin if there's one), otherwise we'll have to see which one is asked for but it's not an immediate dep

zklaus · 2022-03-16T08:58:34Z

@valeriupredoi is right that we should add direct dependencies. Looking at the failing tests here, this includes progressbar, which was added for the new obs cmorization, fire (our command line interface), and mapgenerator.

schlunma · 2022-03-16T09:06:08Z

Sounds good. Three questions/comments on this:

Should we also update our environment.yml with these new packages in the future?
Can we also list dependencies in meta.yml that are not available on conda-forge? As far as I can tell mapgenerator is not available on conda-forge (mamba search mapgenerator doesn't return anything).
I think the packages iris-ued and pyamg are wrongly mentioned by the bot. I'm not sure why we would need a package for Ultrafast electron scattering data exploration 😅

valeriupredoi · 2022-03-16T09:13:37Z

I think meta should contain only conda-forge packages, unless we add a - pip: section to it - @zklaus would know more, but I think we should add all the direct conda-forge deps to our environment.yml too and see also if we can find any more of the pip ones that are on conda-forge/ Yup, fairly sure those two "deps" are picked up wrongly by the bot

valeriupredoi · 2022-03-16T09:17:46Z

agh why do people keep reintroducing numba? It's a PITA

schlunma · 2022-03-16T10:05:29Z

@zklaus can you briefly comment on that? Shall I add an - pip: section to meta.yml for mapgenerator?

zklaus · 2022-03-16T10:07:46Z

I think yes. I am now adding mapgenerator to conda-forge, but I cannot promise a timeline due to dependency on conda-forge people, so we should move ahead with mapgenerator in a pip section for now and try to move it to conda-forge.

Are any of the other packages missing?

schlunma · 2022-03-16T10:09:01Z

Sounds great, thanks! As far as I can tell no other package is missing on conda-forge.

conda-forge-linter · 2022-03-16T10:21:08Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

Failed to even lint the recipe, probably because of a conda-smithy bug 😢. This likely indicates a problem in your meta.yaml, though. To get a traceback to help figure out what's going on, install conda-smithy and run conda smithy recipe-lint . from the recipe directory.

schlunma · 2022-03-16T10:34:13Z

Using conda smithy recipe-lint . locally tells me . is in fine form...

Any ideas? From the failing tests I guess the - pip: section is the problem.

schlunma · 2022-03-16T11:38:20Z

I guess we have to wait until mapgenerator is avaialble on conda-forge: conda/conda-build#548...

conda-forge-linter · 2022-03-16T11:39:54Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

zklaus · 2022-03-16T13:08:51Z

Sometimes the grinders grind fast: conda-forge/staged-recipes#18357 is on the way. We still have to wait ~2hr for things to trickle through.

valeriupredoi · 2022-03-16T13:11:17Z

man that was fast! Excellent work, Klaus! What's with that Proj crappe in the docker build though? prob goes away when Manu removes the pip installation of that package

schlunma · 2022-03-16T13:14:40Z

Awesome, thanks Klaus! Yes, that should go away when I remove the pip install ... again.

schlunma · 2022-03-16T17:21:53Z

mapgenerator is still not available, but the build finished successfully 40 min ago. Will tackle this tomorrow.

schlunma · 2022-03-17T09:51:43Z

I just created a fresh environment and just installed xesmf (mamba install xesmf). Running import xesmf runs fine there. In our environment this fails with the error above. I checked the environments: in the esmvaltool one, numba is missing; in the fresh one it is not missing...weird.

fresh.txt
esmvaltool.txt

schlunma · 2022-03-17T11:22:10Z

I just tested the CDS-UERRA CMORizer with xesmf=0.6.2 (had to manually install numba and update sparse before that). The CMORizer now fails with a

ValueError: To reuse weights, you need to provide either filename or weights.

Full traceback


2022-03-17 10:54:22,409 UTC [19871] ERROR   Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/mnt/lustre02/work/bd0854/b309141/repos/ESMValCore/esmvalcore/_main.py", line 499, in run
    fire.Fire(ESMValTool())
  File "/mnt/lustre02/work/bd0854/b309141/mambaforge/envs/esm/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/mnt/lustre02/work/bd0854/b309141/mambaforge/envs/esm/lib/python3.10/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/mnt/lustre02/work/bd0854/b309141/mambaforge/envs/esm/lib/python3.10/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/mnt/lustre02/work/bd0854/b309141/repos/ESMValTool/esmvaltool/cmorizers/data/cmorizer.py", line 489, in format
    self.formatter.format(start, end, install)
  File "/mnt/lustre02/work/bd0854/b309141/repos/ESMValTool/esmvaltool/cmorizers/data/cmorizer.py", line 194, in format
    if not self.format_dataset(dataset, start, end, install):
  File "/mnt/lustre02/work/bd0854/b309141/repos/ESMValTool/esmvaltool/cmorizers/data/cmorizer.py", line 284, in format_dataset
    success = self._run_pyt_script(in_data_dir, out_data_dir, dataset,
  File "/mnt/lustre02/work/bd0854/b309141/repos/ESMValTool/esmvaltool/cmorizers/data/cmorizer.py", line 386, in _run_pyt_script
    module.cmorization(in_dir, out_dir, cmor_cfg, self.config, start, end)
  File "/mnt/lustre02/work/bd0854/b309141/repos/ESMValTool/esmvaltool/cmorizers/data/formatters/datasets/cds_uerra.py", line 266, in cmorization
    _regrid_dataset(in_dir, var, cfg)
  File "/mnt/lustre02/work/bd0854/b309141/repos/ESMValTool/esmvaltool/cmorizers/data/formatters/datasets/cds_uerra.py", line 236, in _regrid_dataset
    regridder = xe.Regridder(input_ds,
  File "/mnt/lustre02/work/bd0854/b309141/mambaforge/envs/esm/lib/python3.10/site-packages/xesmf/frontend.py", line 779, in __init__
    super().__init__(grid_in, grid_out, method, input_dims=input_dims, **kwargs)
  File "/mnt/lustre02/work/bd0854/b309141/mambaforge/envs/esm/lib/python3.10/site-packages/xesmf/frontend.py", line 318, in __init__
    raise ValueError('To reuse weights, you need to provide either filename or weights.')
ValueError: To reuse weights, you need to provide either filename or weights.

Thus, we have to pin xesmf=0.3.0 for now. I tested this in an installation from source (with xesmf=0.3.0 in environment.yml) and the CMORizer runs fine. The output data is also identical to the old version. 👍

valeriupredoi · 2022-03-17T11:51:53Z

great work, Manu! I can open an issue about that cmorizer if that's OK with you 👍

valeriupredoi · 2022-03-17T11:57:20Z

it seems there is something fishy going on with their latest (xesmf) noarch package, if you look carefully at the conda forge page https://anaconda.org/conda-forge/xesmf the last linux64 supported version is 0.2.0 from ages ago then they switched to a noarch 0.3.0 (2yrs+ old) that we are currently using then they upped it incrementally but looking at the number of downloads 0.3.0 is by far the most used, so I am guessing a few other deps are using that one too

valeriupredoi

awesome work, guys!

schlunma · 2022-03-17T12:11:26Z

@conda-forge-admin, please rerender

github-actions · 2022-03-17T12:13:18Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/esmvaltool-suite-feedstock/actions/runs/1998491111.

zklaus

Very nice work! I was stuck in meetings before, but this looks good to me. I'll chase the xesmf issue upstream; perhaps something for a bugfix or simply the next release.

zklaus · 2022-03-17T12:35:36Z

See pangeo-data/xESMF#158 for the xesmf issue. Having said that, it is a bit odd that we end up pulling in such an ancient version of sparse. We get 0.3.1, which was the very first conda build at the beginning of 2018, now we ware at sparse 0.13 and things have changed a fair bit...

schlunma · 2022-03-17T12:44:41Z

Awesome, thanks for all your help on this @zklaus @valeriupredoi !! 🎉

schlunma · 2022-03-17T15:30:35Z

Having said that, it is a bit odd that we end up pulling in such an ancient version of sparse. We get 0.3.1, which was the very first conda build at the beginning of 2018, now we ware at sparse 0.13 and things have changed a fair bit...

Apparently we actually don't need sparse. I just did a mamba create -n esm "python=3.10" esmvaltool and I got an environment without sparse. The same is true for a development installation of ESMValTool.

However, if I do a mamba install xesmf in the development environment (after manual removal of the PyPi package), I get:

Transaction

  Prefix: /home/manuel/mambaforge/envs/esm

  Updating specs:

   - xesmf
   - ca-certificates
   - certifi
   - openssl


  Package                Version  Build         Channel                  Size
───────────────────────────────────────────────────────────────────────────────
  Install:
───────────────────────────────────────────────────────────────────────────────

  + cf_xarray              0.7.0  pyhd8ed1ab_0  conda-forge/noarch     Cached
  + importlib_metadata    4.11.3  hd8ed1ab_0    conda-forge/noarch     Cached
  + sparse                 0.3.1  py_0          conda-forge/noarch     Cached
  + xarray              2022.3.0  pyhd8ed1ab_0  conda-forge/noarch     Cached
  + xesmf                  0.6.2  pyhd8ed1ab_0  conda-forge/noarch     Cached

  Summary:

  Install: 5 packages

  Total download: 0 B

So it's actually xesmf that pulls in that old version of sparse...

zklaus · 2022-03-17T15:32:46Z

Well, yes, kind of. It is xesmf that pulls sparse, but xesmf has only sparse as a dependency with no mention of the version number. So why don't we get the latest sparse (0.13.*)? There must be something else in the dependency tree from our side conflicting.

schlunma · 2022-03-17T15:47:09Z

Indeed, mamba create -n xesmf "python=3.10" xesmf pulls in sparse=0.13.0.

But since none of our dependencies (and their dependencies, etc.) depends on sparse (otherwise it would be in our environment, right?) none of these dependencies can put a pin to sparse, right? The only way I can think of how this leads to a conflict is that we (or one of our dependencies) put a pin to a common dependency of ours and sparse. Does that make sense?

valeriupredoi · 2022-03-17T16:14:29Z

my knowledge is rather sparse (pun intended) on this sparse package - I see it's an add-on on top of scipy and numpy sparse arrays, but can't see it inside the install env. At any rate, I reckon we good for now, but we really must do some cleaning up for the next release namely get rid of esmvalgroup channel, that still shows up as top search in a Google search for "esmvaltool conda forge" and in the conda-forge search obv https://anaconda.org/search?q=esmvaltool

valeriupredoi · 2022-03-17T16:17:26Z

I will also install the new released version on JASMIN tomorrow, it'd be good if you Manu or @remi-kazeroni pinged the DKRZ IT boffins to install the new version on DKRZ too 👍

zklaus · 2022-03-17T16:26:17Z

I think I found the bugger, but it is not very enlightening: numba limits numpy to < 1.22. Consequently, the solver finds it cheaper to install an older version of sparse without numba that doesn't require a downgrade of numpy. Or so the theory goes...

valeriupredoi · 2022-03-17T16:32:22Z

OK gents conda installation works well (in principle) https://github.com/ESMValGroup/ESMValTool/runs/5588309376?check_suite_focus=true - I have installed the new released 2.5.0 myself with mamba install -c conda-forge esmvaltool and I have 2.5.0 all nicely, but recipe_python.yml crashes at diag level with

2022-03-17 16:26:35,966 [2325] DEBUG    esmvaltool.diag_scripts.shared.plot._plot,331   Creating 'plot' plot None
Traceback (most recent call last):
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/esmvaltool/diag_scripts/examples/diagnostic.py", line 107, in <module>
    main(config)
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/esmvaltool/diag_scripts/examples/diagnostic.py", line 101, in main
    plot_diagnostic(cube, output_basename, provenance_record, cfg)
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/esmvaltool/diag_scripts/examples/diagnostic.py", line 61, in plot_diagnostic
    quickplot(cube, **cfg['quickplot'])
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/esmvaltool/diag_scripts/shared/plot/_plot.py", line 334, in quickplot
    plot_function(cube, **kwargs)
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/iris/quickplot.py", line 294, in plot
    result = iplt.plot(*args, **kwargs)
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/iris/plot.py", line 1564, in plot
    return _draw_1d_from_points("plot", _plot_args, *args, **kwargs)
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/iris/plot.py", line 817, in _draw_1d_from_points
    result = draw_method(u, v, *args, **kwargs)
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/matplotlib/pyplot.py", line 2840, in plot
    return gca().plot(
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/matplotlib/axes/_axes.py", line 1743, in plot
    lines = [*self._get_lines(*args, data=data, **kwargs)]
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/matplotlib/axes/_base.py", line 273, in __call__
    yield from self._plot_args(this, kwargs)
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/matplotlib/axes/_base.py", line 394, in _plot_args
    self.axes.xaxis.update_units(x)
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/matplotlib/axis.py", line 1468, in update_units
    self.set_units(default)
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/matplotlib/axis.py", line 1540, in set_units
    self._update_axisinfo()
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/matplotlib/axis.py", line 1507, in _update_axisinfo
    self.set_default_intervals()
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/matplotlib/axis.py", line 2236, in set_default_intervals
    xmin = self.converter.convert(valmin, self.units, self)
  File "/home/valeriu/miniconda3/lib/python3.9/site-packages/nc_time_axis/__init__.py", line 288, in convert
    ut = cftime.utime(cls.standard_unit, calendar=first_value.calendar)
AttributeError: module 'cftime' has no attribute 'utime'

~~can any of you try reproduce this? I am trying to get the other way of stuff installed - conda env etc and see if I can reproduce the issue~~

EDIT: nevermind, all good! locally fussed conda crappe! We're in business! 🍺

valeriupredoi · 2022-03-17T16:39:13Z

I think I found the bugger, but it is not very enlightening: numba limits numpy to < 1.22. Consequently, the solver finds it cheaper to install an older version of sparse without numba that doesn't require a downgrade of numpy. Or so the theory goes...

ah the golden rule of the solver: downgrade as little as possible, ideally upgrade always/when possible - good find, Klaus!

schlunma · 2022-03-17T16:47:17Z

can any of you try reproduce this? I am trying to get the other way of stuff installed - conda env etc and see if I can reproduce the issue

2022-03-17 16:46:22,352 UTC [2870] INFO    Run was successful

on my local machine!

schlunma · 2022-03-17T16:47:52Z

Ahh, saw the edit too late, phewwww 😄

valeriupredoi · 2022-03-17T16:55:49Z

yeah sorry, but good to check anyway! Many congrats Manu and the Release Boys 🍺

zklaus · 2022-07-20T11:41:56Z

great work, Manu! I can open an issue about that cmorizer if that's OK with you +1

@valeriupredoi, did you ever get around to opening that issue? I didn't find it when I was trying to upgrade that cmorizer at ESMValGroup/ESMValTool#2728 and wanted to make sure I don't duplicate.

valeriupredoi · 2022-07-20T12:22:57Z

@zklaus a very good point - this went under the rug big time, cheers for reminding me about it! No duplication 😁

bouweandela · 2022-07-21T08:14:41Z

That would have saved me an hour of debugging..

regro-cf-autotick-bot added 2 commits March 15, 2022 18:16

updated v2.5.0

ab85ee8

MNT: Re-rendered with conda-build 3.21.8, conda-smithy 3.18.0, and co…

9c092ba

…nda-forge-pinning 2022.03.15.10.12.29

regro-cf-autotick-bot requested review from schlunma, bouweandela, nielsdrost, valeriupredoi and zklaus as code owners March 15, 2022 18:17

Updated dependecies

28c856b

Updated dependecies

b615944

Remove cmorize_obs entry point

33010ed

Removed mapgenerator dependency

4b5e529

Installed mapgenerator from pip

c9df90d

schlunma added 2 commits March 17, 2022 12:23

Pinned xesmf=0.3.0

acaea2a

Removed old cmorize_obs call

d271558

valeriupredoi approved these changes Mar 17, 2022

View reviewed changes

zklaus approved these changes Mar 17, 2022

View reviewed changes

zklaus mentioned this pull request Mar 17, 2022

Dependency refinement sparse, numba pangeo-data/xESMF#158

Closed

schlunma merged commit 7a139be into conda-forge:main Mar 17, 2022

regro-cf-autotick-bot deleted the 2.5.0_hebbfbd branch March 17, 2022 12:51

sloosvel mentioned this pull request Jul 20, 2022

Update development infrastructure ESMValGroup/ESMValTool#2663

Merged

8 tasks

esmvaltool-suite v2.5.0 #5

esmvaltool-suite v2.5.0 #5

Conversation

regro-cf-autotick-bot commented Mar 15, 2022 • edited by schlunma Loading

Pending Dependency Version Updates

Dependency Analysis

Packages found by source code inspection but not in the meta.yaml:

Packages found in the meta.yaml but not found by source code inspection:

conda-forge-linter commented Mar 15, 2022

schlunma commented Mar 16, 2022

Packages found by source code inspection but not in the meta.yaml:

valeriupredoi commented Mar 16, 2022

zklaus commented Mar 16, 2022

schlunma commented Mar 16, 2022

valeriupredoi commented Mar 16, 2022 • edited Loading

valeriupredoi commented Mar 16, 2022

schlunma commented Mar 16, 2022

zklaus commented Mar 16, 2022

schlunma commented Mar 16, 2022

conda-forge-linter commented Mar 16, 2022

schlunma commented Mar 16, 2022

schlunma commented Mar 16, 2022

conda-forge-linter commented Mar 16, 2022

zklaus commented Mar 16, 2022

valeriupredoi commented Mar 16, 2022

schlunma commented Mar 16, 2022

schlunma commented Mar 16, 2022

schlunma commented Mar 17, 2022 • edited Loading

schlunma commented Mar 17, 2022

valeriupredoi commented Mar 17, 2022

valeriupredoi commented Mar 17, 2022

valeriupredoi left a comment

Choose a reason for hiding this comment

schlunma commented Mar 17, 2022

github-actions bot commented Mar 17, 2022

zklaus left a comment

Choose a reason for hiding this comment

zklaus commented Mar 17, 2022

schlunma commented Mar 17, 2022

schlunma commented Mar 17, 2022

zklaus commented Mar 17, 2022

schlunma commented Mar 17, 2022

valeriupredoi commented Mar 17, 2022

valeriupredoi commented Mar 17, 2022

zklaus commented Mar 17, 2022

valeriupredoi commented Mar 17, 2022 • edited Loading

valeriupredoi commented Mar 17, 2022

schlunma commented Mar 17, 2022

schlunma commented Mar 17, 2022

valeriupredoi commented Mar 17, 2022 • edited Loading

zklaus commented Jul 20, 2022

valeriupredoi commented Jul 20, 2022

bouweandela commented Jul 21, 2022

regro-cf-autotick-bot commented Mar 15, 2022 •

edited by schlunma

Loading

valeriupredoi commented Mar 16, 2022 •

edited

Loading

schlunma commented Mar 17, 2022 •

edited

Loading

valeriupredoi commented Mar 17, 2022 •

edited

Loading

valeriupredoi commented Mar 17, 2022 •

edited

Loading