Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Qualification tool: Parsing Execs to get the ExecInfo #2 #5426

Merged
merged 3 commits into from
May 5, 2022

Conversation

nartal1
Copy link
Collaborator

@nartal1 nartal1 commented May 5, 2022

This fixes #5361. It is a follow on PR of #5420.

Execs supported in this PR:

  • GlobalLimitExec
  • LocalLimitExec
  • GenerateExec
  • CartesianProductExec
  • SortExec

Have added eventlog for testing GlobalLimit and LocalLimitExec. These execs are not present in final physical plan if collect is called. So the eventlog is generated using show (df.limit(10).show). For rest of the execs, we create eventlogs on thr fly in the test itself.

Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@nartal1 nartal1 added the tools label May 5, 2022
@nartal1 nartal1 added this to the May 2 - May 20 milestone May 5, 2022
@nartal1 nartal1 requested a review from tgravescs May 5, 2022 01:56
@nartal1 nartal1 self-assigned this May 5, 2022
@nartal1
Copy link
Collaborator Author

nartal1 commented May 5, 2022

build

Copy link
Collaborator

@tgravescs tgravescs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we adding a global_local_limit_eventlog, I thought those execs were already in the nds_q86_test eventlog? I would much prefer just to reuse an existing log

Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@nartal1
Copy link
Collaborator Author

nartal1 commented May 5, 2022

why are we adding a global_local_limit_eventlog, I thought those execs were already in the nds_q86_test eventlog? I would much prefer just to reuse an existing log

GlobalLimit and LocalLimit are in LogicalPlan and not in PhysicalPlan of nds_q86_test eventlog. So I couldn't use that eventlog for testing those execs.

@nartal1
Copy link
Collaborator Author

nartal1 commented May 5, 2022

build

1 similar comment
@nartal1
Copy link
Collaborator Author

nartal1 commented May 5, 2022

build

Copy link
Collaborator

@tgravescs tgravescs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please compress ./tools/src/test/resources/spark-events-qualification/global_local_limit_eventlog using zstd and update path in test

Signed-off-by: Niranjan Artal <nartal@nvidia.com>
@tgravescs
Copy link
Collaborator

build

@nartal1 nartal1 merged commit c1aebfc into NVIDIA:branch-22.06 May 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Qualification tool implement Plan Parsing Execs group1
2 participants