Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add job commit time to task tracker stats #3345

Merged
merged 3 commits into from
Aug 31, 2021

Conversation

jlowe
Copy link
Member

@jlowe jlowe commented Aug 30, 2021

Fixes #3237.

Adds job and task commit time metrics to the columnar write stats tracker classes. This corresponds to the changes in apache/spark@a96e9e197e.

Signed-off-by: Jason Lowe <jlowe@nvidia.com>
@jlowe jlowe added this to the Aug 30 - Sept 10 milestone Aug 30, 2021
@jlowe jlowe self-assigned this Aug 30, 2021
@jlowe
Copy link
Member Author

jlowe commented Aug 30, 2021

build

@tgravescs
Copy link
Collaborator

18:28:49 error file=/home/jenkins/agent/workspace/jenkins-rapids_premerge-github-2458/tests/src/test/scala/com/nvidia/spark/rapids/ParquetWriterSuite.scala message=org.apache.spark. is in wrong order relative to org.apache.spark.internal.io.FileCommitProtocol. line=27 column=0

Copy link
Collaborator

@tgravescs tgravescs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, other then scalastyle error

@jlowe
Copy link
Member Author

jlowe commented Aug 31, 2021

build

revans2
revans2 previously approved these changes Aug 31, 2021
Copy link
Collaborator

@revans2 revans2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one minor nit that I can live with.

sql-plugin/src/main/scala/com/nvidia/spark/Utils.scala Outdated Show resolved Hide resolved
@jlowe
Copy link
Member Author

jlowe commented Aug 31, 2021

build

@jlowe jlowe merged commit 254ced5 into NVIDIA:branch-21.10 Aug 31, 2021
@jlowe jlowe deleted the job-commit-time branch August 31, 2021 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BasicColumnWriteJobStatsTracker might be affected by spark change SPARK-34399
3 participants