New repomap data format

[pirat89]

short

• Introduce processing of the new repomap data (JSON format) in all related actors.
• The implementation is hacky as we expect to significantly change several actors during the next quarter.
• Drop/Deprecate original RepositoriesMap format and introduce the new RepositoriesMapping model
• Fix several newly discovered bugs
• Extend original unit-tests to cover new functionality and expectations by tests
• Inform user when old repomap data file is present to let them know the new data is required (or used in case the RH insights is available)
• Add new mapping logic: multiple repositories from the same repository family will refer just to one repository per target repository family (read more below)
• Only repomap data of the matching version format (1.0.0) is processed . In case a repomap data with a different version format is loaded, ends with error.

Names of methods/models/functions/.. could be bad. Any suggestions are welcomed

Why we want this

The previous repomap data in the CSV format limited us significantly to implement any better logic around the repositories mapping. We would like to provide a possibilty to:

• specify target channels - the original mapping had to contain for various reasons double maps in case of premium channels (e.g. an EUS src repoid -> EUS + GA target repoid; etc). We have been told we should point ideally just to right channels and e.g. in case of E4S we should enable just E4S channels. as well, there are situations when you want to upgrade with EUS repositories but you have only GA repositories available on the source system (typical for the last minor release of RHEL where only GA repositories are present usually)....
• map correctly PES ids from the pes data to repoids - the PES data refers just to PES IDs regarding repositories. So it couldn't decided correctly what repoid should really be available regarding the used system and picks just one the mathis the pesid. This results e.g. in request to enable "rhui" repositories on non-rhui systems, etc.
• The new format is easier for verification and maintainance than the original CSV (orig: 360+ lines of stuff impossible to validate simply; refers to several wrong lines we discovered in the original data during this work)

Incompatible changes

• The original RepositoriesMap model is going to be dropped or deprecated (@oamg/developers point of discussion)
  • The deprecation means we have to add the decision making library into the scanner actor, however the result could be potentially inaccurate; as well, we do not plan to consume this msg anywhere else. The current private library can provide the result simply for us.
  • The drop means that actors consuming this msg will be broken immediately. However, we do not expect anyone else consuming this particular data (even when undocumented expectations..). This is the current expactation to do, but keeping it opened one more time to discuss this.

Why the new introduced model instead of update of the original one?

We want to make people immediately aware they are expecting a message which is different and their code is broken. With the same model name they would realize that problem just after they execute their code (means e.g. in the middle of upgrade). With the new model name, leapp reports the problem immediately when loading leapp repositories.

The new mapping logic

The original solution mapped 1 repoid to X target repoids. In case of non-ga channels, we maped to premium channel + ga channel always. This resulted in real life mapping like:

        src repos       || dst repos
        ----------------||---------------------
        EUSbase         || EUSbaseos, GAbaseos, EUSappstream, GAappstream
        EUSbase, GAbase || EUSbaseos, GAbaseos, EUSappstream, GAappstream
        GAbase          || GAbase, GAappstream
        ...

The new logic provides mapping from one repository family (== repositories with the same PES ID) to target repository families, where each such a family ends with right one repoid in the end (remember mapping rhel 7 base to rhel 8 basos + appstream).
Using the previous table, the new result looks like this:

        src repos       || dst repos
        ----------------||---------------------
        EUSbase         || EUSbaseos, EUSappstream
        EUSbase, GAbase || EUSbaseos, EUSappstream
        GAbase          || GAbase, GAappstream
        ...

In this case the priority of various channels is this: beta < ga < all alse (where all else includes EUS, E4S, .... premium channels). It's not expected to have avilable several premium channels on one system right now and we do not know any expectations about possible different priorities in this case. In case the mapping for particular source repoid and channel doesn't exist, use as fallback a default channel(s). Such a fallback always includes the 'ga' channel right now. Such a case points to two possible root causes:

• the original channel doesn't exist for the target system
• or just the particular expected repository is not provided
• or the most probably - incorrect input repomap data

---

Tasks:

• update repositories_mapping actor
• update setuptargetrepos actor
• update peseventsscanner actor
  - sync with @vinzenz because of conflicting changes regarding the modularity PR The Modularity integration #672
  - simplify the actor -> do not map pesid to repoid inside the actor, work just with pesids
• update repositoriesblacklist actor
• update targetuserspacecreater actor
• create a lib with the repomap translation logic and add it to BL., PESscanner, and setuptargetrepos actors
• drop old models OR add the decision logic into the repomap scanner actor and just deprecate old repos..
• deprecate models we expect to remove in the spring release so people are aware (need to be discussed on cabal yet, probably this could go into an another PR)
• update unit tests
  • RepositoriesBlacklist
  • SetupTargetRepos (original unit tests)
  • SetupTargetRepos tests with different channels
  • PESEventScanner
  • RepositoriesMappingScanner
  • two missing tests repositoriesmappingscanner + update the code to handle loading of missing/empty/invalidformat data
• polish commits

[github-actions]

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergable.
If you want to re-run tests or request review, you can use following commands as a comment:

• leapp-ci build to run copr build and e2e tests in OAMG CI
• review please to notify leapp developers of review request

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please consider rerunning the CI by commenting leapp-ci build (might require several comments). If the problem persists, contact leapp-infra.

[pirat89]

@MichalHe i've polished commits. so all others should be create ideally as fixups etc. to make autorebase possible :) Fixed some bugs, making linter happier, etc. I've started the manual testing and I need to check the differences between original and current results to discover possible issues and we can continue with tests.

[pirat89]

Some outputs regarding the current implementaion. Summary:

• Several issues fixed (e.g. mapping to rhui on non-rhui system...)
• RepositoriesSetupTasks is not produced by peseventscanner now - empty? we should check that
  • before pointed mainly to rhui repos again
  • the requested repositories are not mapped to the target repos now; investigate
• the blacklisted repos should be filterered ideally to skip repos irrelevant to the system
• PESRpmTransactionTasks the install field is now empty with the new repo mapping (bug)

====== old ==================
Stamp: 2021-09-11T13:39:25.714519Z
Actor: repositories_blacklist
Phase: FactsCollection
Type: RepositoriesBlacklisted
Message_data:
{
    "repoids": [
        "codeready-builder-for-rhel-8-x86_64-rpms"
    ]
}
----------------------------------------------------------------------
Stamp: 2021-09-11T13:39:30.429531Z
Actor: setuptargetrepos
Phase: FactsCollection
Type: TargetRepositories
Message_data:
{
    "custom_repos": [],
    "rhel_repos": [
        {
            "repoid": "rhel-8-for-x86_64-baseos-rpms"
        },
        {
            "repoid": "rhel-8-for-x86_64-appstream-rpms"
        },
        {
            "repoid": "rhui-codeready-builder-for-rhel-8-x86_64-rhui-rpms"
        },
        {
            "repoid": "rhui-rhel-8-for-x86_64-appstream-rhui-rpms"
        },
        {
            "repoid": "rhui-rhel-8-for-x86_64-baseos-rhui-rpms"
        }
    ]
}
----------------------------------------------------------------------
Stamp: 2021-09-11T13:39:30.300201Z
Actor: pes_events_scanner
Phase: FactsCollection
Type: RepositoriesSetupTasks
Message_data:
{
    "to_enable": [
        "rhui-codeready-builder-for-rhel-8-x86_64-rhui-rpms",
        "rhui-rhel-8-for-x86_64-appstream-rhui-rpms",
        "rhui-rhel-8-for-x86_64-baseos-rhui-rpms"
    ]
}
----------------------------------------------------------------------
Stamp: 2021-09-11T13:39:30.286469Z
Actor: pes_events_scanner
Phase: FactsCollection
Type: PESRpmTransactionTasks
Message_data:
{
    "local_rpms": [],
    "to_install": [
        "authselect-compat",
        "coreutils-common",
        "dbus-common",
        "dbus-daemon",
        "dbus-tools",
        ....
            ],
    "to_keep": [],
    "to_remove": [
        "GeoIP",
        "Red_Hat_Enterprise_Linux-Release_Notes-7-en-US",
        "aic94xx-firmware",
        ...
    ],
    "to_upgrade": []
}
----------------------------------------------------------------------
----------------------------------------------------------------------
======= new ======================
Stamp: 2021-09-11T13:47:24.915075Z
Actor: repositories_blacklist
Phase: FactsCollection
Type: RepositoriesBlacklisted
Message_data:
{
    "repoids": [
        "codeready-builder-beta-for-rhel-8-s390x-rpms",
        "codeready-builder-beta-for-rhel-8-ppc64le-rpms",
        "rhui-codeready-builder-for-rhel-8-x86_64-rhui-rpms",
        "codeready-builder-beta-for-rhel-8-x86_64-rpms",
        "codeready-builder-for-rhel-8-aarch64-rpms",
        "codeready-builder-for-rhel-8-s390x-rpms",
        "codeready-builder-beta-for-rhel-8-aarch64-rpms",
        "codeready-builder-for-rhel-8-rhui-rpms",
        "codeready-builder-for-rhel-8-x86_64-rhui-rpms",
        "codeready-builder-for-rhel-8-x86_64-rpms",
        "codeready-builder-for-rhel-8-ppc64le-rpms"
    ]
}
----------------------------------------------------------------------
Stamp: 2021-09-11T13:47:32.284552Z
Actor: setuptargetrepos
Phase: FactsCollection
Type: TargetRepositories
Message_data:
{
    "custom_repos": [],
    "rhel_repos": [
        {
            "repoid": "rhel-8-for-x86_64-appstream-rpms"
        },
        {
            "repoid": "rhel-8-for-x86_64-baseos-rpms"
        }
    ]
}
----------------------------------------------------------------------
Stamp: 2021-09-11T13:47:31.643661Z
Actor: pes_events_scanner
Phase: FactsCollection
Type: PESRpmTransactionTasks
Message_data:
{
    "local_rpms": [],
    "to_install": [],
    "to_keep": [],
    "to_remove": [
        "GeoIP",
        "Red_Hat_Enterprise_Linux-Release_Notes-7-en-US",
        ...
            ],
    "to_upgrade": []
}
----------------------------------------------------------------------

btw, the log output:

--- {"message": "2021-09-11 15:47:31.550 WARNING  PID: 20567 leapp.workflow.FactsCollection.pes_events_scanner: Cannot find suitable repository for PES ID: rhel8-appstream", "level": "WARNING"}
--- {"message": "2021-09-11 15:47:31.564 WARNING  PID: 20567 leapp.workflow.FactsCollection.pes_events_scanner: Cannot find suitable repository for PES ID: rhel8-baseos", "level": "WARNING"}
--- {"message": "2021-09-11 15:47:31.581 WARNING  PID: 20567 leapp.workflow.FactsCollection.pes_events_scanner: Cannot find suitable repository for PES ID: rhel8-crb", "level": "WARNING"}

[pirat89]

The current implementation of peseventsscanner provides almost same results as before, except the packages from excluded/blacklisted repositories. In the original implementation such a packages have been suggested to be installed even when they should be excluded as the actor could (and many times did) translate a mapped PES ID as a different repoid (because the pesid represents family of repositories and the original solution could not refer to the correct one, so based on an order of items, basically a random repoid from the family has been taken).

[pirat89]

created PR for the update of e2e tests. However for now we will have to deal with broken e2e tests until it gets to the stable branch.

[pirat89]

updated description of the PR.

[pirat89]

After a short disussion I decided for now to drop the old models completely. We will discuss the final decision about the drop/deprecation the next week.

[MichalHe]

Finished reading the code introduced in this PR. The code is readable and reasonable (but that might be me starting to be really familiar with it 😁), but we need to get rid of the FIXME/TODOs.

[pirat89]

@MichalHe almost complete review. I am in the middle of the repomap library unit tests.

[pirat89]

@MichalHe just one nitpick, but coverage seems pretty good to me. 💯 let's wait for other reviewers.

[pirat89]

@Rezney @vinzenz some changes applied. pls re-review. Btw, one important note, the current split of commits result in broken state of the repository until the very last one. However, regarding the nature of the change, it could be one collasal commit with veeery long commit msg otherwise, so I wanted to go this way to make the changes more readable understandable.

[Rezney]

I suggest to change also this StopActorExecutionError which mentions the old repomap.csv file:

raise StopActorExecutionError(
            message='There are no enabled target repositories for the upgrade process to proceed.',
            details={'hint': (
                'Ensure your system is correctly registered with the subscription manager and that'
                ' your current subscription is entitled to install the requested target version {version}.'
                ' In case the --no-rhsm option (or the LEAPP_NO_RHSM=1 environment variable is set)'
                ' ensure the custom repository file is provided regarding the documentation with'
                ' properly defined repositories or in case repositories are already defined'
                ' in any repofiles under /etc/yum.repos.d/ directory, use the --enablerepo option'
                ' for leapp. Also make sure "/etc/leapp/files/repomap.csv" file is up-to-date.'
                ).format(version=api.current_actor().configuration.version.target)
            }
        )

[pirat89]

I suggest to change also this StopActorExecutionError which mentions the old repomap.csv file:

raise StopActorExecutionError(
            message='There are no enabled target repositories for the upgrade process to proceed.',
            details={'hint': (
                'Ensure your system is correctly registered with the subscription manager and that'
                ' your current subscription is entitled to install the requested target version {version}.'
                ' In case the --no-rhsm option (or the LEAPP_NO_RHSM=1 environment variable is set)'
                ' ensure the custom repository file is provided regarding the documentation with'
                ' properly defined repositories or in case repositories are already defined'
                ' in any repofiles under /etc/yum.repos.d/ directory, use the --enablerepo option'
                ' for leapp. Also make sure "/etc/leapp/files/repomap.csv" file is up-to-date.'
                ).format(version=api.current_actor().configuration.version.target)
            }
        )

This is going to be handled as part of #731 PR - good catch

[pirat89]

@vinzenz @Rezney finito

[pirat89]

@vinzenz @Rezney final finito

[Rezney]

@examon the tests seem to be stuck, could you please take a look?

[Rezney]

There is an ongoing outage which is most probably affecting the rest of the tests. In any way, I performed the e2e test manually and it passed.

[pirat89]

Discussed with @Rezney , merging without latest results to unblock us in other tasks. Anyway, e2e tests passed in the past and we changed just texts recently.