Fix incorrect lscpu parsing

[dkubek]

Original solution expected always key: val pair on each line. However, it has not been expected that val could be actually empty string. Hence the following snippet:

NUMA node1 CPU(s):   
Flags:               fpu vme de pse tsc msr pae mce cx8 ...

has been parsed as:

{
  'NUMA node1 CPU(s)':   'Flags:               fpu vme de pse tsc msr pae mce cx8 ...'
}

instead of:

{
  'NUMA node1 CPU(s)': '',
  'Flags': 'fpu vme de pse tsc msr pae mce cx8 ...'
}

The new solution updates the parsing for output on RHEL 7, but also calls newly lscpu -J on RHEL 8+ to obtain data in the JSON format, which drops all possible parsing problems from our side.

Fixes #1182

[github-actions]

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

• review please @oamg/developers to notify leapp developers of the review request
• /packit copr-build to submit a public copr build using packit

Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build. If you need a different version of leapp from PR#42, use /packit test oamg/leapp#42

It is possible to schedule specific on-demand tests as well. Currently 2 test sets are supported, beaker-minimal and kernel-rt, both can be used to be run on all upgrade paths or just a couple of specific ones.
To launch on-demand tests with packit:

• /packit test --labels kernel-rt to schedule kernel-rt tests set for all upgrade paths
• /packit test --labels beaker-minimal-8.9to9.3,kernel-rt-8.9to9.3 to schedule kernel-rt and beaker-minimal test sets for 8.9->9.3 upgrade path

[Deprecated] To launch on-demand regression testing public members of oamg organization can leave the following comment:

• /rerun to schedule basic regression tests using this pr build and latest upstream leapp build as artifacts
• /rerun 42 to schedule basic regression tests using this pr build and leapp*PR42* as artifacts
• /rerun-sst to schedule sst tests using this pr build and latest upstream leapp build as artifacts
• /rerun-sst 42 to schedule sst tests using this pr build and leapp*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra.

[pirat89]

@dkubek that's actually much better than mine idea 💯 I was thinking what could be the best way having whitespaces without NL, and honestly it didn't come to my mind to negate it in this way.

[dkubek]

I have found yet another case where even this does not work. If there are two empty fields one after another like this:

...
NUMA node1 CPU(s):   80-159
NUMA node2 CPU(s):
NUMA node3 CPU(s):
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
...

the Flags will be parsed like this:

{

    '\nFlags': 'fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs',

}

I have already fixed it, but can you please look at the regex to check with me whether it is sound?

[pirat89]

good catch! I will give it a time when I finish current tasks. At monday latest.

[pirat89]

hmmm. on my machine, I've relized that flags can be multiline

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  12
  On-line CPU(s) list:   0-11
Vendor ID:               GenuineIntel
  Model name:            Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz
    CPU family:          6
    Model:               165
    Thread(s) per core:  2
    Core(s) per socket:  6
    Socket(s):           1
    Stepping:            2
    CPU(s) scaling MHz:  19%
    CPU max MHz:         5100.0000
    CPU min MHz:         800.0000
    BogoMIPS:            5399.81
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx 
                         fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bt
                         s rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vm
                         x smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadli
                         ne_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb s
                         tibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep
                          bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dth
                         erm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp vnmi pku ospke md_clear flush_l1d 
                         arch_capabilities
Virtualization features: 
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   192 KiB (6 instances)
  L1i:                   192 KiB (6 instances)
  L2:                    1.5 MiB (6 instances)
  L3:                    12 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-11

This is not problem for RHEL 8+ where we use the JSON for the parsing. but it's still broken in such a case on for RHEL 7. Question is, whether rhel7 can print also multiline output or not. Also, trying the current regexp, I see that lines with key: \n (empty value) are skipped - such a key is not present in the result dict:

>>> import re
>>> from pprint import pprint as pp
>>> ls_re = re.compile(r'^(?P<name>[^:]+):[^\S\n]+(?P<value>.+)\n?', flags=re.MULTILINE)
>>> lsout3 = '\nsomething:    val val1\ncpu: \ncpu2: something\nflags:  good line jedna another one flag something_flag\nkey3: val3\nflags2: something somthingy\n'
>>> pp(dict(ls_re.findall(lsout3)))
{'\nsomething': 'val val1',
 ' \ncpu2': 'something',
 'flags': 'good line jedna another one flag something_flag',
 'flags2': 'something somthingy',
 'key3': 'val3'}

not sure whether to consider this as a problem or not honestly (speaking about skipped keys).

[pirat89]

about the multiline, seems that /proc/cpuinfo is never multiline.

[pirat89]

I guess we can discuss it on cabal to see how much we want to pay attention to fix the parsing on rhel7. checking the json output solution, seems it works as expected. Tried also on my current system, it seems good for me there.

@oamg/developers headsup ^

[pirat89]

@oamg/developers so we do not need to take care about the multiline. when the lscpu cmd is called via the run function, the executed shell is understand as non-interactive, hence the lscpu does not generate any multilines.

[fernflower]

To make codespell happy we'd need to exclude the test files with --exclude-file option https://github.com/oamg/leapp-repository/blob/master/.github/workflows/codespell.yml#L30

[pirat89]

@dkubek update please also commit msg so it reflects all changes and problem description. I've updated the description of the PR already.

Also please rebase againt up-to-date master to fix some of crashing tests.

[fernflower]

lgtm

[pirat89]

lgtm