Connect MACRO explicitly to MESSAGEix Scenarios

[gidden]

This PR brings explicit MACRO support into the message_ix library. Given a solved MESSAGE-stand alone scenario and additional input data (example xlsx file), MACRO is parameterized and calibrated. There is an additional option to check that calibration against the number of iterations of a MESSAGE-MACRO scenario, checking that n_iter == 1 .

cc @volker-krey @danielhuppmann @ClaraLuisa (and any others? feel free to suggest)

Depends on iiasa/ixmp#177

PR checklist

• explicit GAMS code review - I had to touch the GAMS code in a number of places and would like an explicit review there
  • Aside from Update handling of discount/interest rates in M-M link #319, the only change seems to be the replacement of total_cost and replacement with COST_NODAL_NET in one location. If problems arise, hopefully this helps isolate them.—@khaeru
• figure out how to exclude certain demand sectors. E.g., in the global model, we currently do not include nc_biomass in MACRO. Should this be done implicitly by input data exclusion or explicitly via the API?
• test the code on an existing global model baseline
  • this will require migrating existing R data files to the proposed new data format (see above example)
  • I would very much appreciate assistance here, as I need to start prioritizing my time elsewhere
  • If we can get this working, we can add tests explicitly to our nightly CI that runs over global model instances
• address non-convergence issues
  • in the test example, MACRO standalone does not converge before max_iterations (100) is reached
  • perhaps as a symptom of this non-convergence, tests are failing when checking that MESSAGE-MACRO iterations are == 1 after calibration
• Checks added by @francescolovat according to @khaeru's comment:
  • Rebase/merge master to remove merge conflicts. (done by @francescolovat @khaeru)
  • Consider how to combine with Adjust to ixmp Backend/Model refactor #249. (done by @khaeru)
  • Address comments from @behnam-zakeri. (No action; see https://github.com/iiasa/message_ix/pull/223/files#r392681157—@khaeru)
• Tests added.
• Documentation added. And docstrings for several functions, making it understandable for e.g. the new research assistants. See below.
• Release notes updated.

Follow-up

The following items were not completed as part of this PR, and must be addressed before this code can be released with message_ix 3.0:

• docs added, with a tutorial explaining how to add MACRO to an example case (the already-existing test should be used here) and add to Sphinx docs → see message_ix.macro lacks minimal documentation #315, Add a tutorial demonstrating the use of Scenario.add_macro #316.
• understand unit issues. There are a variety of places in the previous R code as well as existing GAMS code where unit conversions are hardcoded (usually going from $B to $T via either / 1000 or / 1e3). → see Address hardcoded units MACRO calibration and M-M linkage #317.
• decide what to do if MACRO-related structure already exists → see Clarify add_macro() behaviour on a non-'clean' Scenario #320.
• add macro calibration to the nightly tests → see Add nightly test of MESSAGE-MACRO calibration & run #318.
• Wrap up the discussion on interest rates and potentially open a new issue. → see Update handling of discount/interest rates in M-M link #319.

[gidden]

Added lots of potential reviewers. I think since this is so core to our main modeling infrastructure, lots of eyes should go on it, if possible.

[gidden]

rebased on current master. still expect 1 test failure from convergence check. @volker-krey is investigating.

[gidden]

@volker-krey we now check for 0 prices and I added a list item above for supporting the exclusion of certain sectors.

[behnam-zakeri]

• test the code on an existing global model baseline

  • this will require migrating existing R data files to the proposed new data format (see above example)
  • I would very much appreciate assistance here, as I need to start prioritizing my time elsewhere
  • If we can get this working, we can add tests explicitly to our nightly CI that runs over global model instances

Thanks @gidden for this excellent piece of work. I can help with this item. The data of global models (R14 and R11) has been removed previously from the R files and migrated to Excel (please see this PR in message_data). The data can be saved in the format you proposed too.

[gidden]

Hey @behnam2015, this would be great! As a first step, I think we could simply check that calibrated values using this branch match those in the existing calibrated MESSAGE-MACRO scenario. Would you mind helping with that?

I think the steps would be:

1. create an excel file input dataset for the R11 model (based on what you write above, I think that should be pretty straightforward)
2. run the "new" calibration
3. check calibrated values

The code to do so would look something like this:

mp = ixmp.Platform()
base = message_ix.Scenario(mp, 'r11 model', 'uncalibrated baseline') # obviously changing model/scenario names
calibrated = message_ix.Scenario(mp, 'r11 model', 'calibrated baseline')

scen = base.clone(scenario='checking calibration')
scen.add_macro('path_to_excel_file') # we might need to start with also `check_convergence=False`

for par in ['aeei', 'grow']:
    obs = scen.par(par)
    exp = calibrated.par(par)
    pd.testing.assert_frame_equal(obs, exp)

@OFR-IIASA could you please let us know the correct model/scenario names to test here (uncalibrated/calibrated)?

[behnam-zakeri]

Thanks @gidden for elaborating and writing the code. I'll test this and share the results with you.

[OFR-IIASA]

@gidden and @behnam2015 I haven't looked at the code, but I assume we only need to use a single scenario. For this use the CD_Links_SSP2_v2 // baseline scenario. This already has macro calibrated values, but I assume there is some cleanup function, i.e. some function to remove any existing macro related parameters and sets. Otherwise, I would suggest to either add this functionality at some point (make new issue) and for the time being just manually remove any macro related parameters.

[behnam-zakeri]

@gidden, if a scenario already has MACRO-related sets, parameters, variables and equations; there will be an error when initializing these items here.

1. To resolve this issue, we can add a check here for each item, e.g., for sets:
   if not s.has_set(key): s.init_set(key, values)
2. On the other hand, if a scenario has MACRO-related data, we may want to remove that data from the scenario before adding new data (in case the existing data is not correct or relevant anymore). So, we may want to remove these items in the first place and add them again.

[gidden]

Hey @behnam2015, good points here. My initial reaction is that if a scenario already has MACRO-related data, then it should be an error to try to "re-add" it. Perhaps we can add an additional function to remove() MACRO related parameters as well.

This is a bit tricky, because I observed that even in the westeros model, some things already existed (e.g., the par historical_gdp), which I believe could be added automatically on the java side of things. Can @danielhuppmann chime in with his thoughts here?

[gidden]

I did a quick test to remove things and then re-add, but got the following error:

self = <message_ix.core.Scenario object at 0x7efd66bfd0f0>, name = 'historical_gdp', idx_sets = ['node', 'year'], idx_names = None

    def init_par(self, name, idx_sets, idx_names=None):
        """Initialize a new parameter.
    
        Parameters
        ----------
        name : str
            Name of the parameter.
        idx_sets : list of str
            Names of sets that index this parameter.
        idx_names : list of str, optional
            Names of the dimensions indexed by `idx_sets`.
        """
>       self._jobj.initializePar(name, *make_dims(idx_sets, idx_names))
E       jpype._jclass.at.ac.iiasa.ixmp.exceptions.IxException: Parameter 'historical_gdp' can only be initialized with MESSAGE standard dimensions!

For posterity, this is done in commit e48d56f on branch test-macro-remove on my remote

[gidden]

@behnam2015, I have made a new branch on my remote called macro-behnam-test, which includes a new try/catch for init_set. Let's see what @danielhuppmann says before we do anything on this PR branch. But in the meantime, you can use macro-behnam-test to start testing the global model (it includes the ignore_node and ignore_sector feature as well).

[behnam-zakeri]

Thanks @gidden. By removing data, I meant only removing the content of a parameter and not the parameter itself. Something like:
key = 'historical_gdp'

if s.has_par(key):
    s.remove_par(key, s.par(key))
else:
    s.init_par(key, idx_sets)

[volker-krey]

The Westeros calibration ends with very small aeei_correction and grow_correction parameters in the order of 1e-7, but at the same time the gdp_scale and demand_scale parameters that measure the change in GDP and demand that MACRO provides to the calibration values are 0.482 and 0.837, respectively (for all years). The initial grow parameter provided to the calibration process via the GDX consists of 3 entries (with four periods incl. base period) while the examples using the R calibration process include 4 entries, one of which is appended in the data file. As a result the growth_correciton parameter calculated in the calibration process only consists of 2 entries which means that the parameter of one period is not calibrated. My suspicion therefore is that this results in an offset as GAMS seems to be able to work with the mismatch in dimensionality somehow. As a quick test of this hypothesis, I would propose to duplicate the last entry of the grow parameter so that four values are provided.

[gidden]

Hey @volker-krey thanks for the analysis! I'm digging into this now. One clarifying question to wrap my head around it. The model periods (where there are decision variables) are 700, 710, and 720 here. Should the grow parameter (and gdp_calibrate?) have additional values in 690 or 730?

If I look at the R file examples, I see that grow and gdp_calibrate both have values from 2010-2110, so I guess it's in the "base year" (i.e., 2010 in the global case, 690 in the westeros case)?

[volker-krey]

Yes, it is 690 that needs to be added. In principle the GAMS code in macro_calibration.gms should reveal what is needed as this is where things are going wrong.

[gidden]

Ok, great, I can look into it. A first test showed that the calibration is still taking 4 iterations, so I suspect there are a few parameters that need values here. Will let you know when I dig further.

[volker-krey]

Looking at the gdp_scale and demand_scale parameters at the end of the calibration is an additional good test. If these are not ~1, there is going to be a demand feedback in an iterated run.

[gidden]

@behnam2015 I have now added support for reading in data with multiple regions and sectors in a923696. It has also been added to the macro-behnam-test branch, so you should be good to go with further testing of the global model.

[gidden]

rebased now, 🤞 tests will pass =)

[khaeru]

Per discussion at the 2020-03-11 MESSAGE meeting, we will merge this PR roughly as-is, and convert the remaining 6 or 7 TODO items to individual follow-up issues, to be assigned and addressed before the next release.

[khaeru]

Opened six issues.