build_powerplants error in tutorial

[Edudiro]

Hello,

I am doing the tutorial run to ensure everything works smoothly and have come upon an error when building the power plants.

I am using the latest version of pypsa-earth and powerplantmatching. I am also using the tutorial config file.

I believe it has something to do with subprocess.py unable to find some executable needed in duke.py from powerplantmatching

Please find below the error message:

INFO:snakemake.logging:[Mon Oct 9 11:43:41 2023]
rule build_powerplants:
input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/all_clean_generators.csv, resources/shapes/gadm_shapes.geojson
output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
log: logs/build_powerplants.log
jobid: 0
benchmark: benchmarks/build_powerplants
reason: Missing output files: resources/powerplants_osm2pm.csv, benchmarks/build_powerplants, resources/powerplants.csv
resources: tmpdir=C:\Users\SE91723\AppData\Local\Temp, mem_mb=500, mem_mib=477
INFO:snakemake.logging:rule build_powerplants:
input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/all_clean_generators.csv, resources/shapes/gadm_shapes.geojson
output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
log: logs/build_powerplants.log
jobid: 0
benchmark: benchmarks/build_powerplants
reason: Missing output files: resources/powerplants_osm2pm.csv, benchmarks/build_powerplants, resources/powerplants.csv
resources: tmpdir=C:\Users\SE91723\AppData\Local\Temp, mem_mb=500, mem_mib=477

INFO:snakemake.logging:
C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\pypsa\networkclustering.py:16: UserWarning: The namespace pypsa.networkclustering is deprecated and will be removed in PyPSA v0.24. Please use pypsa.clustering.spatial instead.
warnings.warn(
This is the repository path: C:\Users\SE91723\Modelos\pypsa\pypsa-earth
Had to go 0 folder(s) up.
C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with
float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with
float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with
float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with
float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
INFO:pypsa.io:Imported network base.nc has buses, lines, transformers
INFO:powerplantmatching.collection:Create combined dataset for GBPT, GCPT, GEO, GGPT, GGTPT, GHPT, GNPT, GPD, GSPT, GWPT
INFO:powerplantmatching.cleaning:Aggregating blocks in data source 'GCPT'.
Traceback (most recent call last):
File "C:\Users\SE91723\Modelos\pypsa\pypsa-earth.snakemake\scripts\tmptvphn884.build_powerplants.py", line 342, in
pm.powerplants(from_url=False, update=True, config_update=config)
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\powerplantmatching\collection.py", line 233, in powerplants
matched = collect(matching_sources, config=config, **collection_kwargs)
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\powerplantmatching\collection.py", line 99, in collect
dfs = parmap(df_by_name, datasets)
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\powerplantmatching\utils.py", line 378, in parmap
return list(map(f, arg_list))
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\powerplantmatching\collection.py", line 76, in df_by_name
return aggregate_units(df, dataset_name=name, config=config)
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\powerplantmatching\cleaning.py", line 447, in aggregate_units
duplicates = pd.concat([duke(df.query("Country == @c")) for c in countries])
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\powerplantmatching\cleaning.py", line 447, in
duplicates = pd.concat([duke(df.query("Country == @c")) for c in countries])
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\site-packages\powerplantmatching\duke.py", line 81, in duke
sub.run(["java", "-version"], check=True, stderr=sub.PIPE, stdout=sub.PIPE)
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\subprocess.py", line 503, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\subprocess.py", line 971, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\subprocess.py", line 1456, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] El sistema no puede encontrar el archivo especificado
[Mon Oct 9 11:43:54 2023]
INFO:snakemake.logging:[Mon Oct 9 11:43:54 2023]
Error in rule build_powerplants:
jobid: 0
input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/all_clean_generators.csv, resources/shapes/gadm_shapes.geojson
output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
log: logs/build_powerplants.log (check log file(s) for error details)

ERROR:snakemake.logging:Error in rule build_powerplants:
jobid: 0
input: networks/base.nc, configs/powerplantmatching_config.yaml, data/custom_powerplants.csv, resources/osm/clean/all_clean_generators.csv, resources/shapes/gadm_shapes.geojson
output: resources/powerplants.csv, resources/powerplants_osm2pm.csv
log: logs/build_powerplants.log (check log file(s) for error details)

RuleException:
CalledProcessError in file C:\Users\SE91723\Modelos\pypsa\pypsa-earth\Snakefile, line 504:
Command 'C:/Users/SE91723/Anaconda3/envs/pypsa-earth/python.exe "C:\Users\SE91723\Modelos\pypsa\pypsa-earth.snakemake\scripts\tmptvphn884.build_powerplants.py"' returned non-zero exit status 1.
File "C:\Users\SE91723\Modelos\pypsa\pypsa-earth\Snakefile", line 504, in __rule_build_powerplants
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\concurrent\futures\thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in file C:\Users\SE91723\Modelos\pypsa\pypsa-earth\Snakefile, line 504:
Command 'C:/Users/SE91723/Anaconda3/envs/pypsa-earth/python.exe "C:\Users\SE91723\Modelos\pypsa\pypsa-earth.snakemake\scripts\tmptvphn884.build_powerplants.py"' returned non-zero exit status 1.
File "C:\Users\SE91723\Modelos\pypsa\pypsa-earth\Snakefile", line 504, in __rule_build_powerplants
File "C:\Users\SE91723\Anaconda3\envs\pypsa-earth\lib\concurrent\futures\thread.py", line 58, in run
Removing output files of failed job build_powerplants since they might be corrupted:
resources/powerplants_osm2pm.csv
WARNING:snakemake.logging:Removing output files of failed job build_powerplants since they might be corrupted:
resources/powerplants_osm2pm.csv
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2023-10-09T114328.287361.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake\log\2023-10-09T114328.287361.snakemake.log

[ekatef]

Hello @Edudiro! Thanks for reporting :)
Agree that the error likely relates to some mismatch in dependencies.
Could you please double-check that Java is installed in your pypsa-earth environment? Does jave --version work?

[Edudiro]

Indeed, I was missing Java in my environment. For some reason it was not included in the environment.yaml file.

I installed openjdk and powerplantmatching is now running smoothly.

Thank you!