[FEA] Add a progress bar in Qualification tool when it is running

[amahussein]

Signed-off-by: Ahmed Hussein (amahussein) a@ahussein.me

fixes #5911

New Behavior

• by default, the Qualification displays a progress-bar in the console stdout.
• Note that the percentage in the progress header is an integer value.
• example output is as follows:

22/07/11 17:39:04 INFO QualificationAppInfo: Parsing Event Log: file:/Users/ahussein/workspace/repos/rapids-profiling-ui/spark-logs/spark-events/app-20210509200722-0001-tpcds100in1
Qual Tool Progress 70% [============================================>                  ] (82 succeeded + 2 failed) / 119
22/07/11 17:39:15 INFO QualificationAppInfo: Total number of events parsed: 110946 for file:spark-events/app-20220210014716-0238
22/07/11 17:39:15 INFO QualificationAppInfo: file:spark-events/app-20220210014716-0238 has App: app-20220210014716-0238
22/07/11 17:39:15 INFO Qualification: Took 21260ms to process file:spark-events/app-20220210014716-0238
22/07/11 17:39:15 INFO QualificationAppInfo: Parsing Event Log: file:spark-events/local-1650905239440
Qual Tool Progress 71% [=============================================>                 ] (83 succeeded + 2 failed) / 119

Changes

• added a new class ConsoleProgressBar similar to org.apache.spark.ui.ConsoleProgressBar
• The PB can be initialized to be updated in a single console line, but this would overlap with other log messages if the LogInfo was enabled and redirected to stdout.

[amahussein]

build

[nartal1]

build

[amahussein]

Some of the thoughts to mention:

• I thought that adding an argument to enable/disable the PB would not add much value.
• Some users may have the log redirected to STDOUT. That's why I opted to disable inlined PB by default. Otherwise, the user will incur the burden of configuring the log.
• This PB can be useful for the Profiler tool as well. There are some potential refactoring to be done, but this was not in the scope of this very issue:
  • If we move all logging/output inside the OutWriter, then we can manage having inline progressBar without overlapping with messages displaying the files being processed.

@viadea and @mattahrens are you guys fine with the behavior of the PB as implemented here?

[amahussein]

build

[amahussein]

The failures in the CI seem to be unrelated to the changes.
see #5991

[mattahrens]

I'm fine with the proposed logic and output for the progress bar 👍

[viadea]

I am fine with this bar too. thanks!

[amahussein]

Thank you @viadea , @mattahrens and @nartal1 for the feedback!
I will wait until the CI gets fixed, then do another build.

[amahussein]

build

[amahussein]

The CI failures caused by join_test.py are not related.
See the submitted issue #6003

[ttnghia]

build

[pxLi]

build

[amahussein]

The CI is now fixed.
@nartal1 Can you give another thumbs up to merge that PR please?

[amahussein]

I made some changes to the consoleProgressBar.
During runtime, it can be used as a holder for various metrics. This gives more flexibility to use the class without being restricted to hard coded counters defined in the implementations.
At the end of execution, it dumps the counters to the console:

Qual Tool Progress 100% [==================================>] (117 succeeded + 0 failed + 2 N/A) / 119
Qual Tool execution time: 59648ms
	Metrics:
	process.success.count = 117
	process.failure.count = 0
	process.NA.count = 2
	execution.total.count = 119

[amahussein]

build