Add gbenchmark for nvtext normalize functions #7668
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davidwendt
added
3 - Ready for Review
Codecov Report
@@ Coverage Diff @@
## branch-0.19 #7668 +/- ##
===============================================
+ Coverage 82.08% 82.47% +0.38%
===============================================
Files 101 101
Lines 17036 17397 +361
===============================================
+ Hits 13984 14348 +364
+ Misses 3052 3049 -3
Continue to review full report at Codecov.
|
vuule
approved these changes
Mar 22, 2021
codereport
approved these changes
Mar 23, 2021
Comment on lines
+34
to
+35
| auto const n_rows = static_cast<cudf::size_type>(state.range(0)); | ||
| auto const max_str_length = static_cast<cudf::size_type>(state.range(1)); |
|
@gpucibot merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference #5696
Creates a gbenchmark for
nvtext::normalize_spaces()andnvtext::normalize_characters()functions.The benchmarks measures various string lengths and number of rows.
I found that
normalize_spaces()is used in haproxy parsing along withextractso having this benchmark helps measure possible performance improvement solutions there.The
normalize_charactersis the same code used as part of thesubword_tokenizer.Since each requires different memory footprint my initial goal for them to share a common benchmark structure did not work out. So the 2 tests are separate gbenchmark test files.
I refactored some of this code to use the more efficient
make_strings_childrenand this improved the performance ofnormalize_spacesby 2-3x.The current subword-tokenizer gbenchmark is also incorporated into the the TEXT_BENCHMARK gbenchmark.