Fix the check of memory (RAM) limits #984

pirat89 · 2022-11-08T16:48:16Z

The checkmem actor was incorrect as the limits have been written in MiBs however the value obtained from /proc/meminfo is in KiBs. So the actual check has been incorrect.

Also the memory limits have been changed since the creation of the actor. Updating the values in KiBs based on the current table:
https://access.redhat.com/articles/rhel-limits

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2139907

github-actions · 2022-11-08T16:48:28Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please to notify leapp developers of review request
/packit copr-build to submit a public copr build using packit

To launch regression testing public members of oamg organization can leave the following comment:

/rerun to schedule basic regression tests using this pr build and leapp*master* as artifacts
/rerun 42 to schedule basic regression tests using this pr build and leapp*PR42* as artifacts
/rerun-all to schedule all tests (including sst) using this pr build and leapp*master* as artifacts
/rerun-all 42 to schedule all tests (including sst) using this pr build and leapp*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please consider rerunning the CI by commenting leapp-ci build (might require several comments). If the problem persists, contact leapp-infra.

MichalHe

I've check that the memory limits match the minimal memory requirements for RHEL systems according to the attached link, and I have verified that the conversion do GiB to KiB is correct, and it indeed is:

1.5GiB = 1024 * 1024 * 1.5 KiB = 1572864 KiB
3.0GiB = 1024 * 1024 * 3.0 KiB = 3145728 KiB

Also, the patch correctly takes into account that /etc/meminfo incorrectly uses kB while the actual value is in KiB (described here), thus the calculations above are correct wrt. units.

I have tested the proposed PR on a RHEL7 VM with 1GB of memory with the following results:

master - preupgrade succeeds, upgrade is not inhibited
6a5b95e5 (this PR) - preupgrade succeeds with an inhibitor - low memory

The produced inhibitor:

Risk Factor: high (inhibitor)
Title: Minimum memory requirements for RHEL 8 are not met
Summary: Memory detected: 1014516 KiB, required: 1572864 KiB

Note that to perform the testing I had to build the package locally, as copr seemed broken, e.g., the offered artifacts from this PR (see https://dashboard.packit.dev/results/copr-builds/518027).
leapp-upgrade-el7toel8-0.17.0-0.el7.noarch leapp-upgrade-el7toel8-deps-0.17.0-0.el7.noarch

LGTM.

viniciusferrao · 2022-11-08T18:05:05Z

I just wanted to thanks for this quick fix. I should expect this to be landed on RHEL 8.7?

MichalHe · 2022-11-09T11:31:14Z

/rerun

github-actions · 2022-11-09T11:36:04Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5027650

github-actions · 2022-11-09T11:59:52Z

Testing Farm request for RHEL-8.6-rhui/5027650 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-09T12:05:27Z

Testing Farm request for RHEL-7.9-rhui/5027650 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

pirat89 · 2022-11-09T12:12:46Z

@viniciusferrao it's too late for RHEL 8.7 (released already). Most likely it's going to be part of the next release (8.8 GA time)

github-actions · 2022-11-09T13:17:48Z

Testing Farm request for RHEL-8.7.0-Nightly/5027650 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-09T13:50:55Z

Testing Farm request for RHEL-8.6.0-Nightly/5027650 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-09T13:54:14Z

Testing Farm request for RHEL-7.9-ZStream/5027650 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-09T14:23:29Z

Testing Farm request for RHEL-7.9-ZStream/5027650 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

viniciusferrao · 2022-11-09T16:17:34Z

@viniciusferrao it's too late for RHEL 8.7 (released already). Most likely it's going to be part of the next release (8.8 GA time)

Totally missed the 8.7 release.

evgeni · 2022-11-10T10:43:50Z

Small question: was it considered to keep min_req_memory in MB (or even use GB?) instead and then do the calculation of KiB in the actor code? With today's machines, writing memory limits as KB is tedious and makes it much harder to compare with documentation (like https://access.redhat.com/articles/rhel-limits) which uses "sane" units (see this PR where Michal went and re-calculated the numbers, just to be sure).

Obviously, not blocking or anything, but a thing to consider?

pirat89 · 2022-11-10T11:13:27Z

Small question: was it considered to keep min_req_memory in MB (or even use GB?) instead and then do the calculation of KiB in the actor code? With today's machines, writing memory limits as KB is tedious and makes it much harder to compare with documentation (like https://access.redhat.com/articles/rhel-limits) which uses "sane" units (see this PR where Michal went and re-calculated the numbers, just to be sure).

Obviously, not blocking or anything, but a thing to consider?

Good point. let me do it

pirat89 · 2022-11-10T12:20:57Z

@MichalHe I've switched units to MiB and disabled linter for those two division lines due to FP errors.

The checkmem actor was incorrect as the limits have been written in MiBs however the value obtained from /proc/meminfo is in KiBs. So the actual check has been incorrect. Also the memory limits have been changed since the creation of the actor. Updating the values in KiBs based on the current table: https://access.redhat.com/articles/rhel-limits The report msg has been updated to print the values in MiB instead of KiB. Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2139907

MichalHe · 2022-11-19T22:26:21Z

/rerun

github-actions · 2022-11-19T22:30:29Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5048132

github-actions · 2022-11-19T22:51:00Z

Testing Farm request for RHEL-8.6-rhui/5048132 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

MichalHe

The upgrade is correctly inhibited wrt. the latest changes when running on a system with insufficient memory:

Risk Factor: high (inhibitor)
Title: Minimum memory requirements for RHEL 8 are not met
Summary: Memory detected: 990 MiB, required: 1536 MiB

The updated memory limits and their conversion to KiB did not change with the recent changes, thus, LGTM.

I will wait on the results of integration tests before proceeding with merge, but I doubt any problems will be encountered.

github-actions · 2022-11-19T22:57:32Z

Testing Farm request for RHEL-7.9-rhui/5048132 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-19T23:22:04Z

Testing Farm request for RHEL-8.7.0-Nightly/5048132 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-20T00:14:25Z

Testing Farm request for RHEL-8.6.0-Nightly/5048132 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-20T00:25:00Z

Testing Farm request for RHEL-7.9-ZStream/5048132 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-20T02:11:07Z

Testing Farm request for RHEL-7.9-ZStream/5048132 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

MichalHe · 2022-11-21T08:33:34Z

/rerun

github-actions · 2022-11-21T08:38:21Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5057030

github-actions · 2022-11-21T08:59:34Z

Testing Farm request for RHEL-8.6-rhui/5057030 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-21T09:05:17Z

Testing Farm request for RHEL-7.9-rhui/5057030 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-21T09:29:15Z

Testing Farm request for RHEL-8.7.0-Nightly/5057030 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-21T10:27:57Z

Testing Farm request for RHEL-8.6.0-Nightly/5057030 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2022-11-21T10:39:59Z

Testing Farm request for RHEL-7.9-ZStream/5057030 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

MichalHe · 2022-11-21T12:27:38Z

I have successfully performed a 7to8 upgrade to make sure that nothing is broken, since a lot of intergration tests were failing - most likely an infra issue since the tests fail on repo download, which has nothing to do with this patch. Since the upgrade went fine, merging.

github-actions · 2022-11-21T12:39:07Z

Testing Farm request for RHEL-7.9-ZStream/5057030 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

## Packaging - Requires cpio (oamg#979) - Requires python3-gobject-base, NetworkManager-libnm (oamg#969) - Bump leapp-repository-dependencies to 9 (oamg#969, oamg#979) ## Upgrade handling ### Fixes - Add leapp RHUI packages to an allowlist to drop confusing reports (oamg#995) - Check only mounted XFS partitions (oamg#1027) - Detect the kernel-core RPM instead of kernel to prevent an error during post-upgrade phases (oamg#1024) - Disable the amazon-id DNF plugin on AWS during the upgrade stage to omit error messages during the upgrade process caused by missing network connection (oamg#990) - Do not create new *pyc files when running leapp after the DNF upgrade transaction (oamg#1017) - Enable upgrades on s390x when /boot is part of rootfs (oamg#991) - Extend the allow list of RHUI clients by azure-sap-apps to omit confusing report (oamg#974) - Filter out PES events unrelated for the used upgrade path and handle overlapping events (oamg#1008) - Fix scan of ceph volumes on systems without ceph-osd (oamg#1011) - Fix scan of ceph volumes when ceph-osd container is not found (oamg#986) - Fix systemd symlinks that become incorrect during the IPU (oamg#972) - Fix the check of memory (RAM) limits (oamg#984) - Fix the upgrade of IBM Z machines configured with ZFCP (oamg#983) - Ignore external accounts in /etc/passwd (oamg#958) - Inhibit the upgrade when entries in /etc/fstab cause overshadowing during the upgrade (oamg#1009) - Prevent leapp failures caused by re-run of leapp in the upgrade initramfs after previous failure, which causes additional confusing error message hiding original bugs (oamg#996) - Prevent the upgrade with RHSM when a baseos and an appstream target repositories are not discovered (oamg#1001) - RHUI(Azure) handle correctly various SAP images (oamg#1037) - Rework the network configuration handling and parse the configuration data properly (oamg#969) - Set RHSM release for non-ga and non-beta channels (oamg#1033) - Use the "grub" library to find the GRUB device (oamg#989) - [IPU 7 -> 8] Detect corrupted grubenv file (oamg#1012) - [IPU 7 -> 8] Ensure that rsyncd stays enabled if it is enabled prior the upgrade(oamg#1043) - [IPU 7 -> 8] Ensure that satellite metapackages are installed after the upgrade (oamg#994) - [IPU 7 -> 8] Ensure the device_cio_free service stays enabled on s390x after the upgrade (oamg#977) - [IPU 7 -> 8] Fixed checks for RHEL SAP IPU 7.9 -> 8.6 (oamg#978) - [IPU 7 -> 8] Fixed migration of ntp to chrony when the ntp service is masked (oamg#966) - [IPU 7 -> 8] Prevent the traceback during migration of sendmail configuration files when the package is not installed (oamg#1041) - [IPU 7 -> 8] Satellite: reindex all related databases to fix issues due to new locales in RHEL 8 (oamg#1007, oamg#1018) - [IPU 7 -> 8] Use the correct domain name in SSSD reports (oamg#1040) - [IPU 8 -> 9] Added checks for RHEL SAP IPU 8.6 -> 9.0 (oamg#978) - [IPU 8 -> 9] CheckVDO: Ask user for the confirmation only on failures and undetermined devices (oamg#961) - [IPU 8 -> 9] Fix the kernel detection during initramfs creation for new kernel on RHEL 9.2+ (oamg#1048) - [IPU 8 -> 9] Fix the upgrade on Azure using RHUI for SAP Apps images (oamg#975) - [IPU 8 -> 9] Handle correctly firewalld version 0.8 (oamg#963) ### Enhancements - Set new upgrade paths (oamg#988): -- RHEL 7.9 -> 8.8, 8.6 (default: 8.8) -- RHEL 8.6 -> 9.0 -- RHEL 8.8 -> 9.2 - Check that used leapp data are valid and compatible with the installed leapp-repository (oamg#1003) - Detect a proxy configuration in YUM/DNF and adjust an error msg on issues caused by the configuration (oamg#914) - Detect and report systemd symlinks that are broken before the upgrade (oamg#972) - Drop obsoleted upgrade paths (oamg#1047) - Improve remediation instructions for packages in unknown repositories (oamg#1010) - Improve the error message to guide users when discovered more space is needed (oamg#956) - Introduce --nogpgcheck option to skip checking of RPM signatures (oamg#910) - Introduced an option to use an ISO file as a target RHEL version content source (oamg#979) - Introduced possibility to specify what systemd services should be enabled/disabled on the upgraded system (oamg#964) - Map the target repositories also based on the installed content (oamg#967) - Provide common information about systemd services (oamg#959) - Register subscribed systems automatically to Red Hat Insights unless --no-insights-register is used (oamg#1000) - Remove obsoleted GPG keys provided by RH after the upgrade to prevent errors (oamg#1022) - Run upgrade process with checking RPM signatures by default (oamg#910, oamg#993, oamg#1025) - Save breadcrumbs results as RHSM facts (oamg#1002) - Small improvements in various reports (oamg#1038, oamg#1039, oamg#1032) - [IPU 8 -> 9] Detect CIFS also when upgrading from RHEL8 to RHEL9 (PR1035) - [IPU 8 -> 9] Detect RoCE on IBM Z machines and check the configuration is safe for the upgrade (oamg#1030) - [IPU 8 -> 9] Enable upgrades of RHEL 8 for SAP HANA to RHEL 9 on ppc64le (oamg#1042) - [IPU 8 -> 9] Improve the handling of blocklisted certificates (oamg#992) ## Additional changes interesting for devels - Started work on bringing up networking inside the upgrade initramfs - currently available for testing and development purposes when LEAPP_DEVEL_INITRAM_NETWORK is set (oamg#960) - Add leapp debug tools to the upgrade initramfs - dracut upgrade module (oamg#997) - Enable disabling of DNF plugins in the dnfplugin library (oamg#990)

## Packaging - Requires cpio (#979) - Requires python3-gobject-base, NetworkManager-libnm (#969) - Bump leapp-repository-dependencies to 9 (#969, #979) ## Upgrade handling ### Fixes - Add leapp RHUI packages to an allowlist to drop confusing reports (#995) - Check only mounted XFS partitions (#1027) - Detect the kernel-core RPM instead of kernel to prevent an error during post-upgrade phases (#1024) - Disable the amazon-id DNF plugin on AWS during the upgrade stage to omit error messages during the upgrade process caused by missing network connection (#990) - Do not create new *pyc files when running leapp after the DNF upgrade transaction (#1017) - Enable upgrades on s390x when /boot is part of rootfs (#991) - Extend the allow list of RHUI clients by azure-sap-apps to omit confusing report (#974) - Filter out PES events unrelated for the used upgrade path and handle overlapping events (#1008) - Fix scan of ceph volumes on systems without ceph-osd (#1011) - Fix scan of ceph volumes when ceph-osd container is not found (#986) - Fix systemd symlinks that become incorrect during the IPU (#972) - Fix the check of memory (RAM) limits (#984) - Fix the upgrade of IBM Z machines configured with ZFCP (#983) - Ignore external accounts in /etc/passwd (#958) - Inhibit the upgrade when entries in /etc/fstab cause overshadowing during the upgrade (#1009) - Prevent leapp failures caused by re-run of leapp in the upgrade initramfs after previous failure, which causes additional confusing error message hiding original bugs (#996) - Prevent the upgrade with RHSM when a baseos and an appstream target repositories are not discovered (#1001) - RHUI(Azure) handle correctly various SAP images (#1037) - Rework the network configuration handling and parse the configuration data properly (#969) - Set RHSM release for non-ga and non-beta channels (#1033) - Use the "grub" library to find the GRUB device (#989) - [IPU 7 -> 8] Detect corrupted grubenv file (#1012) - [IPU 7 -> 8] Ensure that rsyncd stays enabled if it is enabled prior the upgrade(#1043) - [IPU 7 -> 8] Ensure that satellite metapackages are installed after the upgrade (#994) - [IPU 7 -> 8] Ensure the device_cio_free service stays enabled on s390x after the upgrade (#977) - [IPU 7 -> 8] Fixed checks for RHEL SAP IPU 7.9 -> 8.6 (#978) - [IPU 7 -> 8] Fixed migration of ntp to chrony when the ntp service is masked (#966) - [IPU 7 -> 8] Prevent the traceback during migration of sendmail configuration files when the package is not installed (#1041) - [IPU 7 -> 8] Satellite: reindex all related databases to fix issues due to new locales in RHEL 8 (#1007, #1018) - [IPU 7 -> 8] Use the correct domain name in SSSD reports (#1040) - [IPU 8 -> 9] Added checks for RHEL SAP IPU 8.6 -> 9.0 (#978) - [IPU 8 -> 9] CheckVDO: Ask user for the confirmation only on failures and undetermined devices (#961) - [IPU 8 -> 9] Fix the kernel detection during initramfs creation for new kernel on RHEL 9.2+ (#1048) - [IPU 8 -> 9] Fix the upgrade on Azure using RHUI for SAP Apps images (#975) - [IPU 8 -> 9] Handle correctly firewalld version 0.8 (#963) ### Enhancements - Set new upgrade paths (#988): -- RHEL 7.9 -> 8.8, 8.6 (default: 8.8) -- RHEL 8.6 -> 9.0 -- RHEL 8.8 -> 9.2 - Check that used leapp data are valid and compatible with the installed leapp-repository (#1003) - Detect a proxy configuration in YUM/DNF and adjust an error msg on issues caused by the configuration (#914) - Detect and report systemd symlinks that are broken before the upgrade (#972) - Drop obsoleted upgrade paths (#1047) - Improve remediation instructions for packages in unknown repositories (#1010) - Improve the error message to guide users when discovered more space is needed (#956) - Introduce --nogpgcheck option to skip checking of RPM signatures (#910) - Introduced an option to use an ISO file as a target RHEL version content source (#979) - Introduced possibility to specify what systemd services should be enabled/disabled on the upgraded system (#964) - Map the target repositories also based on the installed content (#967) - Provide common information about systemd services (#959) - Register subscribed systems automatically to Red Hat Insights unless --no-insights-register is used (#1000) - Remove obsoleted GPG keys provided by RH after the upgrade to prevent errors (#1022) - Run upgrade process with checking RPM signatures by default (#910, #993, #1025) - Save breadcrumbs results as RHSM facts (#1002) - Small improvements in various reports (#1038, #1039, #1032) - [IPU 8 -> 9] Detect CIFS also when upgrading from RHEL8 to RHEL9 (PR1035) - [IPU 8 -> 9] Detect RoCE on IBM Z machines and check the configuration is safe for the upgrade (#1030) - [IPU 8 -> 9] Enable upgrades of RHEL 8 for SAP HANA to RHEL 9 on ppc64le (#1042) - [IPU 8 -> 9] Improve the handling of blocklisted certificates (#992) ## Additional changes interesting for devels - Started work on bringing up networking inside the upgrade initramfs - currently available for testing and development purposes when LEAPP_DEVEL_INITRAM_NETWORK is set (#960) - Add leapp debug tools to the upgrade initramfs - dracut upgrade module (#997) - Enable disabling of DNF plugins in the dnfplugin library (#990)

pirat89 requested a review from a team November 8, 2022 16:48

pirat89 added the bug Something isn't working label Nov 8, 2022

MichalHe previously approved these changes Nov 8, 2022

View reviewed changes

pirat89 added this to the 8.8/9.2 milestone Nov 9, 2022

pirat89 dismissed MichalHe’s stale review via 7adb575 November 10, 2022 11:20

pirat89 force-pushed the fix-memory-check branch 2 times, most recently from 7adb575 to ba0feb6 Compare November 10, 2022 12:20

pirat89 force-pushed the fix-memory-check branch from ba0feb6 to 19d2b46 Compare November 10, 2022 12:34

MichalHe approved these changes Nov 19, 2022

View reviewed changes

MichalHe merged commit e3936eb into oamg:master Nov 21, 2022

pirat89 added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Jan 20, 2023

pirat89 mentioned this pull request Feb 21, 2023

# Release 0.18.0 #1049

Merged

pirat89 deleted the fix-memory-check branch March 10, 2023 09:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix the check of memory (RAM) limits #984

Fix the check of memory (RAM) limits #984

pirat89 commented Nov 8, 2022

github-actions bot commented Nov 8, 2022

MichalHe left a comment

viniciusferrao commented Nov 8, 2022

MichalHe commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

pirat89 commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

viniciusferrao commented Nov 9, 2022

evgeni commented Nov 10, 2022

pirat89 commented Nov 10, 2022

pirat89 commented Nov 10, 2022

MichalHe commented Nov 19, 2022

github-actions bot commented Nov 19, 2022

github-actions bot commented Nov 19, 2022

MichalHe left a comment

github-actions bot commented Nov 19, 2022

github-actions bot commented Nov 19, 2022

github-actions bot commented Nov 20, 2022

github-actions bot commented Nov 20, 2022

github-actions bot commented Nov 20, 2022

MichalHe commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

MichalHe commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

Fix the check of memory (RAM) limits #984

Fix the check of memory (RAM) limits #984

Conversation

pirat89 commented Nov 8, 2022

github-actions bot commented Nov 8, 2022

Thank you for contributing to the Leapp project!

MichalHe left a comment

Choose a reason for hiding this comment

viniciusferrao commented Nov 8, 2022

MichalHe commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

pirat89 commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

github-actions bot commented Nov 9, 2022

viniciusferrao commented Nov 9, 2022

evgeni commented Nov 10, 2022

pirat89 commented Nov 10, 2022

pirat89 commented Nov 10, 2022

MichalHe commented Nov 19, 2022

github-actions bot commented Nov 19, 2022

github-actions bot commented Nov 19, 2022

MichalHe left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 19, 2022

github-actions bot commented Nov 19, 2022

github-actions bot commented Nov 20, 2022

github-actions bot commented Nov 20, 2022

github-actions bot commented Nov 20, 2022

MichalHe commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

github-actions bot commented Nov 21, 2022

MichalHe commented Nov 21, 2022

github-actions bot commented Nov 21, 2022