Error while reading configspace for BOHB

[Neeratyoy]

Hi,

I have configspace.json in the same directory as the configs.json and results.json outputs from BOHB. However, on running CAVE from CLI, it seemingly tries to use ConfigSpace's read_and_write.pcs_new.py file to read the configspace.json and not read_and_write.json.write from ConfigSpace.
ConfigSpace seemingly has issues with the pcs_new read, write: automl/ConfigSpace#110

However, my directory only contains configspace.json and not configspace.pcs, yet error trace shows:

File "/usr/local/lib/python3.6/dist-packages/ConfigSpace/read_and_write/pcs_new.py", line 145, in build_condition
", ".join(condition.values))
TypeError: sequence item 0: expected str instance, int found

As per load_configspace() in hpbandster2smac.py, shouldn't having a configspace.json instead of configspace.pcs suffice?

Thanks.

[shukon]

Hi, thanks for filing the issue.
It should. I just fixed an issue regarding the use of floats in CategoricalHyperparameters right now, can you check if it's fixed for you on the development-branch?
Otherwise: can you please state the versions of CAVE, SMAC and ConfigSpace you are using? And maybe print the whole (or at least a bit more) of the stacktrace, so I know what part of CAVE is using the wrong pcs-loading-routine?
Cheers.

[shukon]

Actually I don't think it has anything to do with the float-in-categorical-problem. So the stacktrace would be really helpful (or maybe even the debug-log, which you can find under CAVE-output-dir/debug/debug.log).

[Neeratyoy]

The debug file:
debug.log

The stack trace:

INFO:cave.cavefacade.CAVE:Saving results to 'bohb/test/cave/'
INFO:smac.scenario.scenario.Scenario:Output to /tmp/tmp6_7ermk4
Traceback (most recent call last):
File "/usr/local/bin/cave", line 11, in
load_entry_point('cave==1.1.3', 'console_scripts', 'cave')()
File "/usr/local/lib/python3.6/dist-packages/cave/cave_cli.py", line 312, in entry_point
cave.main_cli()
File "/usr/local/lib/python3.6/dist-packages/cave/cave_cli.py", line 292, in main_cli
verbose_level=verbose_level)
File "/usr/local/lib/python3.6/dist-packages/cave/cavefacade.py", line 229, in init
self.bohb_result, folders = HpBandSter2SMAC().convert(folders[0])
File "/usr/local/lib/python3.6/dist-packages/cave/utils/hpbandster2smac.py", line 37, in convert
paths = list(self.hpbandster2smac(result, cs, backup_cs, tmp_dir).values())
File "/usr/local/lib/python3.6/dist-packages/cave/utils/hpbandster2smac.py", line 156, in hpbandster2smac
scenario.write()
File "/usr/local/lib/python3.6/dist-packages/smac/scenario/scenario.py", line 462, in write
self.out_writer.write_scenario_file(self)
File "/usr/local/lib/python3.6/dist-packages/smac/utils/io/output_writer.py", line 55, in write_scenario_file
new_value = self._parse_argument(scenario, key, getattr(scenario, key))
File "/usr/local/lib/python3.6/dist-packages/smac/utils/io/output_writer.py", line 93, in _parse_argument
self.write_pcs_file(scenario.cs, new_path)
File "/usr/local/lib/python3.6/dist-packages/smac/utils/io/output_writer.py", line 160, in write_pcs_file
fh.write(pcs.write(cs))
File "/usr/local/lib/python3.6/dist-packages/ConfigSpace/read_and_write/pcs_new.py", line 462, in write
condition_lines.write(build_condition(condition))
File "/usr/local/lib/python3.6/dist-packages/ConfigSpace/read_and_write/pcs_new.py", line 145, in build_condition
", ".join(condition.values))
TypeError: sequence item 0: expected str instance, int found

Package versions:

> cave.__version__
> '1.1.3'
> smac.__version__
> '0.8.0'
> ConfigSpace.__version__
> '0.4.8'

Hope that helps!
Thanks.

[shukon]

Thanks a lot. There has been changes in the processing of pcs-files in the newer versions of SMAC and CAVE. The SMAC 0.8.0 saved the pcs-file in the new-pcs format. That should be fixed (*) in version 0.10.0. Also, CAVE version 1.1.4 resolved some bugs in the interpretation of the configspace, though that might even be unnecessary when upgrading SMAC to version 0.10.0.

EDIT: (*) fixed means, SMAC now writes the json-format when saving the configspace. This is important, since SMAC is used for an intermediate step when reading in BOHB-files.

[Neeratyoy]

I had to install SMAC 0.8.0 because of the following:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/init.py", line 583, in _build_master
ws.require(requires)
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/init.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/init.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (smac 0.10.0 (/usr/local/lib/python3.6/dist-packages), Requirement.parse('smac==0.8.0'), {'cave'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/cave", line 6, in
from pkg_resources import load_entry_point
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/init.py", line 3191, in
@_call_aside
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/init.py", line 3175, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/init.py", line 3204, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/init.py", line 585, in _build_master
return cls._build_from_requirements(requires)
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/init.py", line 598, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/local/lib/python3.6/dist-packages/pkg_resources/init.py", line 786, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'smac==0.8.0' distribution was not found and is required by cave

[shukon]

Oh you are right - CAVE ver 1.1.3 was not compatible to SMAC >= 0.8.0.
However, CAVE ver 1.1.4 is compatible to (and even requires) SMAC 0.10.0. So I'd encourage you to just update CAVE to version 1.1.4 and see if you can reproduce the error.

[Neeratyoy]

Upgrading CAVE made it finally work. Thanks!

However, it threw this error:

File "ConfigSpace/util.py", line 73, in ConfigSpace.util.impute_inactive_values
File "ConfigSpace/configuration_space.py", line 1035, in ConfigSpace.configuration_space.Configuration.init
File "ConfigSpace/configuration_space.py", line 1061, in ConfigSpace.configuration_space.Configuration.is_valid_configuration
File "ConfigSpace/c_util.pyx", line 38, in ConfigSpace.c_util.check_configuration
File "ConfigSpace/c_util.pyx", line 138, in ConfigSpace.c_util.check_configuration
File "ConfigSpace/configuration_space.py", line 815, in ConfigSpace.configuration_space.ConfigurationSpace._check_forbidden
File "ConfigSpace/c_util.pyx", line 28, in ConfigSpace.c_util.check_forbidden
File "ConfigSpace/c_util.pyx", line 35, in ConfigSpace.c_util.check_forbidden
ConfigSpace.exceptions.ForbiddenValueError: Given vector violates forbidden clause (Forbidden: maxpool_kernel_1 in {5, 6} && Forbidden: kernel_2 in {'5', '7'})

Is it because the EPM samples values that are forbidden?
These forbidden clauses so far haven't thrown any issues while running BOHB and seem to be working fine.

debug.log

[shukon]

Mh - ok so this is very interesting, thanks for bringing that up. The problem is not the epm sampling illegal values. For the ConfiguratorFootprint CAVE projects configurations into a two-dimensional plane, for which inactive values (which are None in ConfigSpace) are imputed by their default-value. However, in your case the default value is illegal at that point, s.t. an Error is raised when using the method ConfigSpace provides. The solution will be to adapt the impute-method to accept forbidden values in CAVE for the given purpose.
This will be fixed, but it might take a couple of days. I will ask you to check again at some point. Thanks.

[Neeratyoy]

Alright. Thanks for the clarification!

[shukon]

Ok, actually this might be solved quicker than expected. Can you please install CAVE from the issue-related branch using e.g. pip install git+https://github.com/automl/CAVE.git@issue-225-forbidden-clauses-cfp and check if it's solved? There will still be an error for the LPI-parameter-importance, but that's to be fixed here. Just run CAVE with the option --parameter_importance fanova until it's fixed there (will leave this issue open until then).

[Neeratyoy]

The branch linked definitely worked better. Gave a CAVE report with more sections than before. Definitely helped. Thanks!

However, the forbidden clause error reappears in the fanova part.
Attaching the debug logs.
debug.log

[shukon]

Oh dear. So I believe, this might better be fixed at ConfigSpace-level. Can you please pip install git+https://github.com/automl/ParameterImportance.git@issue/98-lpi-error-on-forbidden-conditional to see if it fixes all the errors? Since I cannot reproduce the error on fanova, I depend on your feedback here. Thanks a lot for bearing with me :)

EDIT: while this (hopefully) kills the error by just assuming nothing to be forbidden during epm-estimation, we are working on a better solution on pimp's side... will update here if we come up with something.

[Neeratyoy]

Got the following error now.
debug.log

Well, I wasn't able to run CAVE at all earlier and with your help, I can get some report out. Which works for me for now! Thanks a lot.
I think I have enough going for my current work at hand :)

[shukon]

Glad to hear that, thanks a lot for the detailed feedback and feel free to open issues if you run into any other problems. I'll leave this issue open until it's fixed.