Add implicit safeFree for RapidsBuffer
#5214
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: remzi <13716567376yh@gmail.com>
firestarman
reviewed
Apr 12, 2022
| @@ -59,6 +59,30 @@ object RapidsPluginImplicits { | |||
| } | |||
| } | |||
|
|
|||
| implicit class RapidsBufferColumn[A <: RapidsBuffer](rapidsBuffer: RapidsBuffer) { | |||
There was a problem hiding this comment.
Suggested change
| implicit class RapidsBufferColumn[A <: RapidsBuffer](rapidsBuffer: RapidsBuffer) { | |
| implicit class RapidsBufferColumn[A <: RapidsBuffer](rapidsBuffer: A) { |
??
| * | ||
| * @param e Exception which we don't want to suppress | ||
| */ | ||
| def safeFree(e: Throwable = null): Unit = { |
There was a problem hiding this comment.
The freeing is safe only when e is not null. Is it expected ?
sperlingxx
reviewed
Apr 12, 2022
| * Exception was thrown prior to this free, it adds the new exception to the suppressed | ||
| * exceptions, otherwise just throws | ||
| * | ||
| * @param e Exception which we don't want to suppress |
There was a problem hiding this comment.
Does the comment align with the implementation ?
…on (NVIDIA#5202) Signed-off-by: Niranjan Artal <nartal@nvidia.com>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
Signed-off-by: Chong Gao <res_life@163.com>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
[auto-merge] branch-22.04 to branch-22.06 [skip ci] [bot]
…VIDIA#5232) * Fix issue in GpuArrayExists where a parent view outlived the child Signed-off-by: Alessandro Bellina <abellina@nvidia.com> * Remove replaceChildNullsByFalse folding it into the caller
[auto-merge] branch-22.04 to branch-22.06 [skip ci] [bot]
* update log warning Signed-off-by: remzi <13716567376yh@gmail.com> * update log warning Signed-off-by: remzi <13716567376yh@gmail.com> * update log warning Signed-off-by: remzi <13716567376yh@gmail.com>
…ource (NVIDIA#5171) * Fix the bug COALESCING reading does not work for v2 parquet/orc datasource Signed-off-by: Bobby Wang <wbo4958@gmail.com> * add unit tests * resolve conflicts * fix a bug
Signed-off-by: Peixin Li <pxli@nyu.edu>
[auto-merge] branch-22.04 to branch-22.06 [skip ci] [bot]
Signed-off-by: Chong Gao <res_life@163.com>
* Throw again after logging that RMM could not intialize Signed-off-by: Alessandro Bellina <abellina@nvidia.com> * Let exception as is from RMM, and change slightly the wording from the ExecutorPlugin
Signed-off-by: sperlingxx <lovedreamf@gmail.com>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
abellina
reviewed
Apr 15, 2022
| @@ -145,7 +146,7 @@ class RapidsBufferCatalog extends Logging { | |||
| def removeBuffer(id: RapidsBufferId): Unit = { | |||
| val buffers = bufferMap.remove(id) | |||
| if (buffers != null) { | |||
There was a problem hiding this comment.
do we need this null check? It seems the implicit checks that the sequence is not null, so we should be really safe to just do:
val buffers = bufferMap.remove(id)
buffers.safeFree()
Signed-off-by: remzi <13716567376yh@gmail.com>
HaoYang670
changed the title
safeFree for RapidsBuffersafeFree for RapidsBuffer
Signed-off-by: Chong Gao <res_life@163.com>
|
build |
* prototype Signed-off-by: remzi <13716567376yh@gmail.com> * add hex, digit, escape Signed-off-by: remzi <13716567376yh@gmail.com> * add more comments Signed-off-by: remzi <13716567376yh@gmail.com> * add schema gen Signed-off-by: remzi <13716567376yh@gmail.com> * rename functions Signed-off-by: remzi <13716567376yh@gmail.com> * rename file, support multi lines Signed-off-by: remzi <13716567376yh@gmail.com> * temp save Signed-off-by: remzi <13716567376yh@gmail.com> * fuly support json grammar Signed-off-by: remzi <13716567376yh@gmail.com> * add a random test Signed-off-by: remzi <13716567376yh@gmail.com> * fix some bug Signed-off-by: remzi <13716567376yh@gmail.com> * rename Signed-off-by: remzi <13716567376yh@gmail.com> * add copyright Signed-off-by: remzi <13716567376yh@gmail.com> * remove xfail Signed-off-by: remzi <13716567376yh@gmail.com> * wider range of chars Signed-off-by: remzi <13716567376yh@gmail.com> * avoid unexpected escape chars Signed-off-by: remzi <13716567376yh@gmail.com> * delete the test data file Signed-off-by: remzi <13716567376yh@gmail.com> * bring back xfail, because fuzz test might randomly fail Signed-off-by: remzi <13716567376yh@gmail.com> * disable fuzz tests by default Signed-off-by: remzi <13716567376yh@gmail.com> * upadte readme Signed-off-by: remzi <13716567376yh@gmail.com> * store schema for debugging Signed-off-by: remzi <13716567376yh@gmail.com> * remove useless package fix spelling mistake Signed-off-by: remzi <13716567376yh@gmail.com> * Update integration_tests/src/main/python/json_fuzz_test.py Co-authored-by: Niranjan Artal <50492963+nartal1@users.noreply.github.com> Co-authored-by: Niranjan Artal <50492963+nartal1@users.noreply.github.com>
Signed-off-by: Andy Grove <andygrove@nvidia.com>
…uffers (NVIDIA#5268) * Use BaseDeviceMemoryBuffer in RapidsGdsStore to accomodte cudaMalloced buffers Signed-off-by: Alessandro Bellina <abellina@nvidia.com> * Fixed sliced leak
…pendency (NVIDIA#5249) Signed-off-by: Jason Lowe <jlowe@nvidia.com>
Signed-off-by: Raza Jafri <rjafri@nvidia.com> Co-authored-by: Raza Jafri <rjafri@nvidia.com>
* Append my id to blossom-ci whitelist * Signoff Signed-off-by: Anthony Chang <antchang@nvidia.com> Co-authored-by: Anthony Chang <antchang@nvidia.com>
Signed-off-by: Peixin Li <pxli@nyu.edu>
…DIA#5424) * Exclude all unicode line terminator characters from matching dot Signed-off-by: Anthony Chang <antchang@nvidia.com> * Use existing terminatorChars seq, add unit tests Signed-off-by: Anthony Chang <antchang@nvidia.com>
Signed-off-by: Raza Jafri <rjafri@nvidia.com> Co-authored-by: Raza Jafri <rjafri@nvidia.com>
* add test for parquet read Signed-off-by: remzi <13716567376yh@gmail.com> * add csv test Signed-off-by: remzi <13716567376yh@gmail.com> * remove unused import Signed-off-by: remzi <13716567376yh@gmail.com> * remove weird import Signed-off-by: remzi <13716567376yh@gmail.com>
This PR adds tests verifying the fix for correctly dealing with all-null data partitions in rapidsai/cudf#10779. Signed-off-by: Gera Shegalov <gera@apache.org>
* Added a shim for CurrentBatchIterator Signed-off-by: Raza Jafri <rjafri@nvidia.com> * Extracted super class for common code Signed-off-by: Raza Jafri <rjafri@nvidia.com> * removed the duplicate code from child class Signed-off-by: Raza Jafri <rjafri@nvidia.com> * removed redundant code Signed-off-by: Raza Jafri <rjafri@nvidia.com> * addressed review comments Signed-off-by: Raza Jafri <rjafri@nvidia.com> * addressed review comments Signed-off-by: Raza Jafri <rjafri@nvidia.com> * added new line Signed-off-by: Raza Jafri <rjafri@nvidia.com> * removed the xfailing test Signed-off-by: Raza Jafri <rjafri@nvidia.com> Co-authored-by: Raza Jafri <rjafri@nvidia.com>
* Change the key for repo Signed-off-by: Hao Zhu <hazhu@nvidia.com> * Use key package to install cuda repo key Signed-off-by: Hao Zhu <hazhu@nvidia.com> * update cuda to latest Signed-off-by: Hao Zhu <hazhu@nvidia.com> * remove old key Signed-off-by: Hao Zhu <hazhu@nvidia.com>
…ping stage times to sql execs (NVIDIA#5440) * calculate the other time based on stage time * change away from per stage calculation * fixes * fix attempt id * more logging * change way duration execs handled * Add in FileSourcescan * Add BatchScan * add InsertIntoHadoopFsRelationCommand * more reads * move logic parsing * Add separate ReadParser * score read * update BatchScan * start write formatting * debug * finish updating writes * Adding tests scan * update test * fix 0 speedup * update more tests * reenable the reads checks * fix failure checking * add test for batchscan * update test * have batrch scan print format * fix speedup < 1 * start test for InsertIntoHadoopFsRelationCommand * update write test * use lower case * add test for CreateDataSourceTableAsSelectCommand Signed-off-by: Thomas Graves <tgraves@apache.org> * update CreateTable * in memory table scan test and test cleanup * fixes * update Create test * use try finally * change to use eventlog * add event log for create table * fix test * fix test * Add Exchange support * Adding in broadcast and subquery broadcast * debug * Change to get driver accums * fix subquery * fix metric name subquerybroadcast * Add event logs for aqe and custom shuffle * Add custom shuffle reader * update test * fix test * cleanup case statement * fix cast * Refactor stages in sql node * change stages from option * fix flatten * testing * fix merge * cleanup and filter execs should be removed * update exchange duration test * move exchange test * debug * convert nanos to ms * fix timings * comment * fix return types * fix seq returned * fix tests * fix tests * add debug text * add helper * filter children * change wholestagecodegen speedup to remove not supported * debug * fix bug * cleanup * turn logs to debug * update logging * cleanup Signed-off-by: Thomas Graves <tgraves@apache.org> * Review comments * fix exec name * review comments
…#5354) * modify shim layers update tests Signed-off-by: remzi <13716567376yh@gmail.com> * correct the path Signed-off-by: remzi <13716567376yh@gmail.com> * fix compile error Signed-off-by: remzi <13716567376yh@gmail.com> * fix compile error Signed-off-by: remzi <13716567376yh@gmail.com> * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/arithmetic.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/arithmetic.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/arithmetic.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/arithmetic.scala Co-authored-by: Jason Lowe <jlowe@nvidia.com> * resolve conflict Signed-off-by: remzi <13716567376yh@gmail.com> * update the shim layer of 321cdh Signed-off-by: remzi <13716567376yh@gmail.com> Co-authored-by: Jason Lowe <jlowe@nvidia.com>
Signed-off-by: Raza Jafri <rjafri@nvidia.com> Co-authored-by: Raza Jafri <rjafri@nvidia.com>
…5454) * QualificationTool. Add speedup information to AppSummaryInfo Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me> * address review comments Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
…oc. (NVIDIA#5466) Signed-off-by: Hao Zhu <hazhu@nvidia.com>
…free' into issue5098_harden_call_to_buffer_free
HaoYang670
requested review from
jlowe,
revans2,
tgravescs,
GaryShen2008 and
NvTimLiu
as code owners
|
build |
|
Close this PR as something weird happened when merging the branch. I will open a new PR. |
|
open #5471 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: remzi 13716567376yh@gmail.com
Close #5098
Add implicit classes to safely free
RapidsBuffers