Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OderBy output stage spill #7759

Closed
wants to merge 9 commits into from

Conversation

duanmeng
Copy link
Collaborator

@duanmeng duanmeng commented Nov 28, 2023

Add a new type of spiller named kOrderByOutput.
It is used to spill data during the output processing
stage of OrderBy operator. The spiller uses a newly
added API that supports spill data by row pointers.

Copy link

netlify bot commented Nov 28, 2023

Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit cf3a326
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/657180e470d3a70008da75eb

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 28, 2023
@duanmeng duanmeng changed the title add OderBy output stage spill Add OderBy output stage spill Nov 28, 2023
@duanmeng duanmeng force-pushed the obspill branch 7 times, most recently from 4a6013c to b96d687 Compare November 29, 2023 09:02
Copy link
Contributor

@xiaoxmeng xiaoxmeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@duanmeng thanks for the change % minors on code structure.

velox/exec/Spiller.h Outdated Show resolved Hide resolved
/// Invoked to spill.Spill all rows pointed by the pointers in sortedRows.This
/// is only used by 'kOrderByOutput' spiller type to spill during the order by
/// output processing. Similarly, the spilled rows still stays in the row
/// container.The caller needs to erase them from the row container.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/container.The/container. The/

velox/exec/OrderBy.cpp Outdated Show resolved Hide resolved
velox/exec/Spiller.h Show resolved Hide resolved
velox/exec/Spiller.h Outdated Show resolved Hide resolved
velox/exec/Spiller.cpp Outdated Show resolved Hide resolved
velox/exec/Spiller.cpp Outdated Show resolved Hide resolved
velox/exec/Spiller.cpp Outdated Show resolved Hide resolved
velox/exec/Spiller.cpp Outdated Show resolved Hide resolved
@duanmeng duanmeng force-pushed the obspill branch 3 times, most recently from 01583a8 to a981427 Compare November 30, 2023 07:06
@duanmeng
Copy link
Collaborator Author

@xiaoxmeng Hi Xuan, I've made a small refactor and resolved all your comments. Could you please help take another look? Thanks.

@duanmeng duanmeng force-pushed the obspill branch 3 times, most recently from 6b0400a to 551b3b5 Compare November 30, 2023 15:30
Copy link
Contributor

@xiaoxmeng xiaoxmeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@duanmeng LGTM. Could you help to add to support hive sort writer in followup? Thanks!

velox/exec/OrderBy.cpp Outdated Show resolved Hide resolved
velox/exec/OrderBy.cpp Outdated Show resolved Hide resolved
velox/exec/Spiller.cpp Outdated Show resolved Hide resolved
velox/exec/Spiller.cpp Outdated Show resolved Hide resolved
velox/exec/Spiller.cpp Outdated Show resolved Hide resolved
velox/exec/Spiller.h Outdated Show resolved Hide resolved
velox/exec/Spiller.cpp Show resolved Hide resolved
Copy link
Contributor

@xiaoxmeng xiaoxmeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@duanmeng LGTM. Thanks!

velox/exec/SortBuffer.cpp Outdated Show resolved Hide resolved
@duanmeng
Copy link
Collaborator Author

duanmeng commented Dec 7, 2023

@xiaoxmeng Hi Xuan, thanks for your review and guidance, I've resolved all the comments, could you please help to take a look?

Copy link
Contributor

@xiaoxmeng xiaoxmeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@duanmeng thanks!

@facebook-github-bot
Copy link
Contributor

@xiaoxmeng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@xiaoxmeng merged this pull request in ddc3471.

Copy link

Conbench analyzed the 1 benchmark run on commit ddc34718.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

yma11 added a commit to yma11/velox that referenced this pull request Dec 8, 2023
zhztheplayer added a commit to zhztheplayer/velox that referenced this pull request Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants