Add Export to SIA405 Abwasser and VSA-DSS 2015

[sjib]

To import and export data in qwat / QGEP there is an INTERLIS import / export functionality in the form of a python library qgepqwat2ili. In QGEP it is also integrated in the plugin and provides two buttons (Import from INTERLIS / export to INTERLIS).
Currently it is configured to import / export VSA_KEK_2019_LV95 when selecting the parameter qgep and SIA405_Wasser_2015_LV95 when selecting the parameter qwat (only export) (see https://github.com/QGEP/qgepqwat2ili )

The new feature should extend this library to allow the user to select in qgep between those 3 datamodels:
a) VSA_KEK_2019_LV95 as is
b) SIA405_Abwasser_2015_LV95
c) DSS_2015_LV95
SIA405_Abwasser_2015_LV95 is from the matching part already existing, as VSA_KEK_2019_LV95 is an extension of this model. The only difference to a) is, that the KEK classes should not be written and the model HEADER will be different.

For VSA-DSS additional matching has to be added to the existing matching scripts – see https://github.com/QGEP/qgepqwat2ili/tree/master/qgepqwat2ili/qgep especially https://github.com/QGEP/qgepqwat2ili/blob/master/qgepqwat2ili/qgep/mapping.py

It is suggested to have the different models in parallel with a subfolder for the matchings per model and a parameter per model (currently qgep and qwat, new e.g. qgepkek2019, qgepsia405abwasser2015, qgepdss2015, ...). This allows to not have model dependencies and be able to add also model 2020 release to the tool without disturbing already implemented model releases and version.

Currently the config.py has settings for SCHEMA_ABWASSER:

ABWASSER_SCHEMA = "pg2ili_abwasser"
ABWASSER_ILI_MODEL = os.path.join(ILI_FOLDER, "VSA_KEK_2019_2_d_LV95-20210120.ili")
ABWASSER_ILI_MODEL_NAME = "VSA_KEK_2019_LV95"

So if there are three different MODELS ABWASSER the question is how to implement this in the config.py and the import / export code that it is a easy as possible to create a pulldown menu with the different export or import model options and select ABWASSER_SCHEMA, ABWASSER_ILI_MODEL and ABWASSER_ILI_MODEL_NAME based on that.

Possible Options:

• a) Add if statements that ABWASSER_SCHEMA is set based on Model Selection in config.py:

if gui_export.comboBox_modelselection.currentText() == "VSA_KEK_2019_LV95":
ABWASSER_SCHEMA = "pg2ili_abwasser"
ABWASSER_ILI_MODEL = os.path.join(ILI_FOLDER, "VSA_KEK_2019_2_d_LV95-20210120.ili")
ABWASSER_ILI_MODEL_NAME = "VSA_KEK_2019_LV95"

elif gui_export.comboBox_modelselection.currentText() == "SIA405_ABWASSER_2015_LV95":
ABWASSER_SCHEMA = "pg2ili_sia405abwasser"
ABWASSER_ILI_MODEL = os.path.join(ILI_FOLDER, "SIA405_Abwasser_2015_2_d-20180417.ili")
ABWASSER_ILI_MODEL_NAME = "SIA405_ABWASSER_2015_LV95"

How to reference on a new combBox_modelselection.currentText() in class GuiExport(QDialog)

• b) Add Subsection to config.py described as in https://pythontect.com/python-configparser-tutorial/ - would look like a very easy solution if config.VSA_KEK_2019_LV95.ABWASSER_SCHEMA instead of config.ABWASSER_SCHEMA would work
• c) Make selection based on chosen model in code - seems pretty a lot of work as config.ABWASSER_SCHEMA is used many times in the code.

@olivierdalang Any thoughts on this?

[sjib]

Implementing VSA-DSS 2015 LV95 needs a newer version of ili2pg, as else there is a conflict in writing the T_ILI2DB_COLUMN_PROP table ((value too long for type character varying(1024), as there are all tables of VSA-DSS listed in the dataset )

see also claeis/ili2db#374, fixed in ili2db 4.5.0

ili2db 4.5.0 (2021-03-12)
------------------------------
..
- size of setting columns in metatables increased to 10240 (#374)

Attention: qgepwat2ili does not use anymore the version specified in the config.py but the one of the installed (now required) modelbaker version - see #71

[sjib]

Export of SIA405_ABWASSER_LV95 directly from VSA_KEK_2019_LV95

As VSA_KEK_2019_LV95 is an extension of SIA405_ABWASSER_LV95 it is possible to directly export SIA405_ABWASSER_LV95 from the KEK-Schema.

I tested with the following command, if an export of SIA405_ABWASSER_2015_LV95 out of the pg2ili_abwasser schema is possible:

java -jar C:/Daten/Dropbox/in_Arbeit/QGEP/Interlis_Schnittstelle_ili2pg/ili2pg-4.9.0/ili2pg-4.9.0.jar --export --modeldir %ilimodelpath% --models SIA405_ABWASSER_2015_LV95 --exportModels SIA405_ABWASSER_2015_LV95 --skipReferenceErrors --sqlEnableNull --createEnumTabs --createFk --noSmartMapping --defaultSrsAuth EPSG --defaultSrsCode 2056 --dbhost %host% --dbport %port% --dbdatabase %db% --dbschema pg2ili_abwasser --dbusr %user% --dbpwd %pwd% --log casse_SIA405_Abwasser_from_KEK.log --trace casse_SIA405_Abwasser_from_KEK.xtf

In case of exporting SIA405_ABWASSER_LV95 the existing code has to be modified or a second export statement has to be added using these parameters: --models SIA405_ABWASSER_2015_LV95 --exportModels SIA405_ABWASSER_2015_LV95

qgepqwat2ili/qgepqwat2ili/__init__.py

Line 133 in 49b419f

utils.ili2db.export_xtf_data(SCHEMA, ILI_MODEL_NAME, args.path, make_log_path(log_path, "iliexport"))

and the function export_xtf_data(schema, model_name, xtf_file, log_path) needs an additional optional argument for the parameter --exportModels

export_xtf_data(schema, model_name, export_model_name, xtf_file, log_path)

qgepqwat2ili/qgepqwat2ili/utils/ili2db.py

Line 60 in 49b419f

def export_xtf_data(schema, model_name, xtf_file, log_path):

and/or and option to make the call with --exportModels

qgepqwat2ili/qgepqwat2ili/utils/ili2db.py

Line 64 in 49b419f

f'"{config.JAVA}" -jar "{config.ILI2PG}" --export --models {model_name} --dbhost {pgconf["host"]} --dbport {pgconf["port"]} --dbusr {pgconf["user"]} --dbpwd {pgconf["password"]} --dbdatabase {pgconf["dbname"]} --dbschema {schema} --modeldir "{config.ILI_FOLDER}" --disableValidation --skipReferenceErrors --createTidCol --noSmartMapping --defaultSrsCode 2056 --log "{log_path}" --trace "{xtf_file}"'

Add --exportModels{model_name} if additional SIA405_ABWASSER_LV95 export

    `f'"{config.JAVA}" -jar "{config.ILI2PG}" --export --models {model_name} --exportModels{model_name} --dbhost {pgconf["host"]} --dbport {pgconf["port"]} --dbusr {pgconf["user"]} --dbpwd {pgconf["password"]} --dbdatabase {pgconf["dbname"]} --dbschema {schema} --modeldir "{config.ILI_FOLDER}" --disableValidation --skipReferenceErrors --createTidCol --noSmartMapping --defaultSrsCode 2056 --log "{log_path}" --trace "{xtf_file}"'`

Maybe a if statement would be the solution: if export_model_name not empty / Null / "" then A else B

Testexport data with log:
casse_SIA405_Abwasser_from_KEK.zip

[sjib]

Import of SIA405_ABWASSER_LV95 directly to VSA_KEK_2019_LV95

As VSA_KEK_2019_LV95 is an extension of SIA405_ABWASSER_LV95 it should be possible to directly import SIA405_ABWASSER_LV95 into the KEK-Schema.

I tested with the following command, if an import of SIA405_ABWASSER_2015_LV95 to the the pg2ili_abwasser schema is possible:

java -jar C:/Daten/Dropbox/in_Arbeit/QGEP/Interlis_Schnittstelle_ili2pg/ili2pg-4.9.0/ili2pg-4.9.0.jar --import --deleteData --trace --sqlEnableNull --skipReferenceErrors --createEnumTabs --createFk --defaultSrsCode 2056 --noSmartMapping --dbhost %host% --dbport %port% --dbdatabase %db% --dbschema pg2ili_abwasser --dbusr %user% --dbpwd %pwd% --log datenimport_pg2ili_abwasser_casse_SIA405_Abwasser_from_KEK.log C:/Daten/TEKSI/4_Feature_requests/QGEP_INTERLIS_Improvements/casse_SIA405_Abwasser_from_KEK.xtf

It worked out well so I just selected the same file from with the INTERLIS Import button and it also worked.

To display the data correctly and be able to do a network following downstream / upstream I had to run the main_cover and main_node scripts (see #75) and refresh the network topology manually.