-
Notifications
You must be signed in to change notification settings - Fork 62
NGen Tutorial
This Tutorial will walk through the steps to build NGen and execute several formulations on a small example catchment. We will use a terminal to download, compile, and run the NGen framework.
A few dependencies are required to build and run NGen, those are documented in the dependencies document. For this tutorial, the minimum required dependencies are C/C++ compiler, CMake, and Boost. Additionally, to use the Fortran BMI adapter, a Fortran Compiler is required to to build the middleware, gfortran should be sufficient.
The first step after ensuring your environment has the required dependencies is to clone the NGen repository.
git clone https://github.com/noaa-owp/ngen
cd ngenFor this tutorial, we are going to use several external components, and we need to make sure they are properly set up.
- First, make sure all the project submodules have been retrieved/updated.
git submodule update --init --recursive
- It is possible to do this for an individual submodule as well, by appending
-- <submodule_path>to the above command. The submodules paths can be found in the.gitmodulesfile.
- Obtain any needed external BMI modules. Several external modules have had configurations to build shared library files added within the ngen repo. Those can be built using the following commands (note on some systems you may need to use
cmake3instead ofcmake).cmake -B extern/cfe/cmake_build -S extern/cfe/cfe/ -DNGEN=ON make -C extern/cfe/cmake_build cmake -B extern/topmodel/cmake_build -S extern/topmodel make -C extern/topmodel/cmake_build cmake -B extern/noah-owp-modular/cmake_build -S extern/noah-owp-modular -DNGEN_IS_MAIN_PROJECT=ON make -C extern/noah-owp-modular/cmake_build cmake -B extern/evapotranspiration/cmake_build -S extern/evapotranspiration/evapotranspiration/ make -C extern/evapotranspiration/cmake_build cmake -B extern/sloth/cmake_build -S extern/sloth make -C extern/sloth/cmake_build
- These examples are for convenience; depending on your realization configuration, you may not need to build all or any of these.
- What matters is that the external module location (meaning the shared library file path for C/C++/Fortran) matches the realization configuration.
- To use the noah-owp-modular fortran library, we need to build the iso_c_fortran_bmi middleware
cmake -B extern/iso_c_fortran_bmi/cmake_build -S extern/iso_c_fortran_bmi make -C extern/iso_c_fortran_bmi/cmake_build
- While this is necessary at the time of this writing, this step should no longer be needed once Issue #333 is resolved.
NGen builds are managed by the CMake build system. To create a build configuration to compile NGen, use a command like the following (assumed to be run from within the ngen repo directory):
cmake -B cmake_build -S . -DNGEN_WITH_PYTHON:BOOL=OFF -DNGEN_WITH_BMI_FORTRAN:BOOL=ON -DUDUNITS_QUIET:BOOL=ONThis configures NGen to compile without Python support, and with Fortran BMI support, which is needed for Noah-OWP respectively. Note that Python support in particular must be explicitly set to OFF to deactivate it.
If you need Python, you can set it to ON, but you may need to create a local virtual environment directory at .venv and install numpy.
Tip
-DUDUNITS_QUIET:BOOL=ON will silence any unit conversion warnings that UDUnits emits
Tip
-DNGEN_WITH_SQLITE:BOOL=ON will enable support to directly read/use geopackage hydrofabrics
Now we can compile NGen
make -j 8 -C cmake_buildBefore continuing, it might be a good idea to run the unit tests for NGen.
By default, NGEN_WITH_TESTS:BOOL=ON the unit tests are built automatically, so you can do:
ls cmake_build/testto see what unit test executable are built, the run the tests, such as.
./cmake_build/test/test_unitand the BMI integration tests
./cmake_build/test/test_bmi_c
./cmake_build/test/test_bmi_fortran
./cmake_build/test/test_bmi_python # Only if you activated Python, thoughCopy and paste the following into your shell. If you are on a system with both cmake and cmake3, set an alias to make sure you use cmake3.
alias cmake=cmake3Then run
git clone https://github.com/noaa-owp/ngen &&
cd ngen &&
git submodule update --init --recursive &&
cmake -B extern/cfe/cmake_build -S extern/cfe/cfe/ -DNGEN=ON &&
make -C extern/cfe/cmake_build &&
cmake -B extern/topmodel/cmake_build -S extern/topmodel &&
make -C extern/topmodel/cmake_build &&
cmake -B extern/noah-owp-modular/cmake_build -S extern/noah-owp-modular -DNGEN_IS_MAIN_PROJECT=ON &&
make -C extern/noah-owp-modular/cmake_build &&
cmake -B extern/evapotranspiration/cmake_build -S extern/evapotranspiration/evapotranspiration &&
make -C extern/evapotranspiration/cmake_build &&
cmake -B extern/sloth/cmake_build -S extern/sloth &&
make -C extern/sloth/cmake_build &&
cmake -B extern/iso_c_fortran_bmi/cmake_build -S extern/iso_c_fortran_bmi &&
make -C extern/iso_c_fortran_bmi/cmake_build &&
cmake -B cmake_build -S . -DNGEN_WITH_PYTHON:BOOL=ON -DNGEN_WITH_BMI_C:BOOL=ON -DNGEN_WITH_BMI_FORTRAN:BOOL=ON &&
make -j 8 -C cmake_build &&
cmake -B extern/test_bmi_c/cmake_build -S extern/test_bmi_c &&
make -C extern/test_bmi_c/cmake_build &&
cmake -B extern/test_bmi_fortran/cmake_build -S extern/test_bmi_fortran -DNGEN_IS_MAIN_PROJECT=ON &&
make -C extern/test_bmi_fortran/cmake_build &&
./cmake_build/test/test_unit &&
./cmake_build/test/test_bmi_c &&
./cmake_build/test/test_bmi_fortranGet a quick coffee...Then you should see
[ OK ] Bmi_Fortran_Formulation_Test.determine_model_time_offset_0_c (16 ms)
[----------] 11 tests from Bmi_Fortran_Formulation_Test (283 ms total)
[----------] Global test environment tear-down
[==========] 90 tests from 2 test suites ran. (334 ms total)
[ PASSED ] 90 tests.Everything built correctly!
Important
For a more information on the ngen build process, see Building.
Now that NGen is compiled, we can run some simple formulation tests. The input to ngen looks like
ngen <catchment_data_path> <catchment subset ids> <nexus_data_path> <nexus subset ids> <realization_config_path>Important
For a more information on ngen usage, see Usage.
Important
For a more information on ngen's use of formulations/BMI, see Formulations and BMI.
CFE requires a Potential EvapoTranspiration input. This input needs to be supplied from some module. We could use SLoTH to provide a fake value to CFE, or we can use the evapotranspiration BMI model to compute a real PET value. We will use the PET module built in the previous steps to supply values to CFE.
The following commands create a space to store the simulation outputs and then links the relevant directories needed for forcing and libraries.
mkdir cfe
cd cfe
ln -s ../data
ln -s ../externThe realization file configures the relevant BMI settings required to connect to the CFE formulation. We will customize the realization file to configure CFE with PET, copy the following JSON to realization.json in your cfe directory.
Realization Configuration JSON
{
"global": {
"formulations": [
{
"name": "bmi_multi",
"params": {
"model_type_name": "bmi_multi_pet_cfe",
"forcing_file": "",
"init_config": "",
"allow_exceed_end_time": true,
"main_output_variable": "Q_OUT",
"modules": [
{
"name": "bmi_c++",
"params": {
"model_type_name": "bmi_c++_sloth",
"library_file": "./extern/sloth/cmake_build/libslothmodel.so",
"init_config": "/dev/null",
"allow_exceed_end_time": true,
"main_output_variable": "z",
"uses_forcing_file": false,
"model_params": {
"sloth_ice_fraction_schaake(1,double,m,node)": 0.0,
"sloth_ice_fraction_xinanjiang(1,double,1,node)": 0.0,
"sloth_smp(1,double,1,node)": 0.0
}
}
},
{
"name": "bmi_c",
"params": {
"model_type_name": "bmi_c_pet",
"library_file": "./extern/evapotranspiration/cmake_build/libpetbmi",
"forcing_file": "",
"init_config": "./data/bmi/c/pet/{{id}}_bmi_config.ini",
"allow_exceed_end_time": true,
"main_output_variable": "water_potential_evaporation_flux",
"registration_function": "register_bmi_pet",
"variables_names_map": {
"water_potential_evaporation_flux": "potential_evapotranspiration"
},
"uses_forcing_file": false
}
},
{
"name": "bmi_c",
"params": {
"model_type_name": "bmi_c_cfe",
"library_file": "./extern/cfe/cmake_build/libcfebmi",
"forcing_file": "",
"init_config": "./data/bmi/c/cfe/{{id}}_bmi_config.ini",
"allow_exceed_end_time": true,
"main_output_variable": "Q_OUT",
"registration_function": "register_bmi_cfe",
"variables_names_map": {
"water_potential_evaporation_flux": "potential_evapotranspiration",
"atmosphere_air_water~vapor__relative_saturation": "SPFH_2maboveground",
"land_surface_air__temperature": "TMP_2maboveground",
"land_surface_wind__x_component_of_velocity": "UGRD_10maboveground",
"land_surface_wind__y_component_of_velocity": "VGRD_10maboveground",
"land_surface_radiation~incoming~longwave__energy_flux": "DLWRF_surface",
"land_surface_radiation~incoming~shortwave__energy_flux": "DSWRF_surface",
"land_surface_air__pressure": "PRES_surface",
"ice_fraction_schaake" : "sloth_ice_fraction_schaake",
"ice_fraction_xinanjiang" : "sloth_ice_fraction_xinanjiang",
"soil_moisture_profile" : "sloth_smp"
},
"uses_forcing_file": false
}
}
],
"uses_forcing_file": false
}
}
],
"forcing": {
"file_pattern": ".*{{id}}.*..csv",
"path": "./data/forcing/",
"provider": "CsvPerFeature"
}
},
"time": {
"start_time": "2015-12-01 00:00:00",
"end_time": "2015-12-30 23:00:00",
"output_interval": 3600
}
}Finally, run ngen, pointing it to the hydrofabric files and selecting just cat-27 to run.
../cmake_build/ngen data/catchment_data.geojson cat-27 data/nexus_data.geojson nex-26 realization.jsonNow you should see in your directory the outfiles cat-27.csv and nex-26_output.csv. The cat-27.csv file contains a table
of the model's OUTPUT_VARS for each time step. nex-26_output.csv contains the nexus output, which is the modules main_output_variable (as defined in the realization configuration file). This is currently multiplied by the catchments area (meters squared) as determined from the hydrofabric.
Note
This is using a generic config file to initialize the model.
For this model run, we will repeat the basic steps above:
cd ..
mkdir topmod
cd topmod
ln -s ../data
ln -s ../externWe need to edit/create the config file we previously used, so copy it to this directory:
cp ../cfe/realization.json ./realization.jsonNow copy some test data for topmodel
cp ./extern/topmodel/topmodel/data/* dataEdit the config file realization.json, specifically the global configuration.
In the bmi_multi params block, set the following variables to these values:
"model_type_name": "bmi_multi_pet_topmodel""main_output_variable: "Qout"
We don't need the sloth module to run topmodel, so remove the first module in the bmi_multi modules list (the sloth module).
We will continue to use PET module for providing PET values to topmodel.
Finally, we can edit the last bmi_multi module to load topmodel.
Set the following variables to these values:
"model_type_name": "bmi_c_topmodel""library_file": "./extern/topmodel/cmake_build/libtopmodelbmi.<dylib/so>""init_config": "./data/topmod.run""main_output_variable": "Qout""registration_function": "register_bmi_topmodel"
Additionally, TOPMODEL writes its own output files unless specifically told not to. If you want to only have Nextgen output, please set both the stand_alone and output flags to 0 in topmod.run and subcat.dat, respectively. More info here and here and here.
Now run the model
../cmake_build/ngen data/catchment_data.geojson cat-27 data/nexus_data.geojson nex-26 realization.jsonOnce again, you can check the outputs in cat-27.csv and nex-26_output.csv.
Note
Comparing the cat-*.csv output file from the previous CFE model run, you will see the different model output variables from the two models being captured by ngen and written out.
So far, we have configured and run two different model formulations (CFE with PET and SLoTH, and Topmodel with PET). We have applied these formulations in a global configuration, but we have only simulated a single catchment in our domain. Next, we will run a formulation for the entire hydrofabric domain, then we will modify the configuration to run more than one formulation in the domain.
In our topmodel working directory, we will run ngen again, this time on the entire domain.
../cmake_build/ngen data/catchment_data.geojson all data/nexus_data.geojson all realization.jsonNote
Now we are asking ngen to run "all" catchments and associated nexuses, instead of a subset of the hydrofabric.
Now inspecting the outfiles created, we will see cat-27.csv and nex-26_output.csv as before, as well as cat-52.csv, cat-67.csv, nex-34_output.csv, and nex-68_output.csv.
Now we will re-configure the realization to run a heterogeneous model where cat-27 runs a unique formulation.
Setup a space to run the model:
cd ..
mkdir noahowp
cd noahowp
ln -s ../data
ln -s ../externCopy the previous realization config to the directory:
cp ../topmod/realization.json ./realization.jsonNow copy the namelist (model options) for Noah-OWP-Modular:
cp extern/noah-owp-modular/noah-owp-modular/run/namelist.input .Open the newly copied namelist.input file and modify the parameter_dir to include the path for the parameter tables. For this example, that is ./extern/noah-owp-modular/noah-owp-modular/parameters/.
Edit the config file realization.json, we will specify an override formulation for cat-27.
We need to create the "catchments": {} key, and add our catchment specific formulations. Add a comma after the "time" object and then add the following:
"catchments":{
"cat-27":{
"formulations":[
{
"name": "bmi_fortran",
"params": {
"model_type_name": "bmi_fortran_noahowp",
"library_file": "./extern/noah-owp-modular/cmake_build/libsurfacebmi",
"registration_function": "register_bmi",
"init_config": "namelist.input",
"main_output_variable": "QINSUR",
"forcing_file": "",
"allow_exceed_end_time": true,
"uses_forcing_file": false,
"variables_names_map" : {
"PRCPNONC" : "atmosphere_water__liquid_equivalent_precipitation_rate",
"Q2" : "atmosphere_air_water~vapor__relative_saturation",
"SFCTMP" : "land_surface_air__temperature",
"UU" : "land_surface_wind__x_component_of_velocity",
"VV" : "land_surface_wind__y_component_of_velocity",
"LWDN" : "land_surface_radiation~incoming~longwave__energy_flux",
"SOLDN" : "land_surface_radiation~incoming~shortwave__energy_flux",
"SFCPRS" : "land_surface_air__pressure"
}
}
}
],
"forcing": {
"file_pattern": ".*{{id}}.*..csv",
"path": "./data/forcing/",
"provider": "CsvPerFeature"
}
}
}
Notice we also added a catchment specific forcing key to indicate the forcing data to use for this catchment. We used the same file pattern and location as before, but we could also choose to point this catchment to a new source of forcing data if we wanted to.
Now run the model
../cmake_build/ngen data/catchment_data.geojson all data/nexus_data.geojson all realization.jsonCheck out the outputs produced, and you will see that cat-27.csv contains Noah-OWP-Modular output variables, whereas the other catchments contain topmodel outputs! We have successfully run a heterogenous model over our three example catchments!
Tutorial
Getting Started
Configuration
Technical References