Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-29378][R] Upgrade SparkR to use Arrow 0.15 API #26555

Closed
wants to merge 6 commits into from
Closed

[SPARK-29378][R] Upgrade SparkR to use Arrow 0.15 API #26555

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3519b23
AppVeyor test
dongjoon-hyun Nov 16, 2019
25c46a0
0.15.1 test
dongjoon-hyun Nov 16, 2019
b486093
Update API
dongjoon-hyun Nov 16, 2019
a4dc812
Revert unneeded
dongjoon-hyun Nov 16, 2019
ea5304d
Address comment
dongjoon-hyun Nov 16, 2019
96f20f8
Update RecordBatchStreamReader
dongjoon-hyun Nov 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions R/pkg/R/SQLContext.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,9 +166,9 @@ writeToFileInArrow <- function(fileName, rdf, numPartitions) {
for (rdf_slice in rdf_slices) {
batch <- arrow::record_batch(rdf_slice)
if (is.null(stream_writer)) {
stream <- arrow::FileOutputStream(fileName)
stream <- arrow::FileOutputStream$create(fileName)
schema <- batch$schema
stream_writer <- arrow::RecordBatchStreamWriter(stream, schema)
stream_writer <- arrow::RecordBatchStreamWriter$create(stream, schema)
Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Nov 16, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ARROW-5505 changes their R API. cc @srowen , @HyukjinKwon , @felixcheung

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shaneknapp . It seems that our Jenkins doesn't have R Arrow installation and skip these test.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not Windows/JDK8 issue. Arrow 0.15 has a breaking change in R interface.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shaneknapp . It seems that our Jenkins doesn't have R Arrow installation and skip these test.

this is correct. should i infer that we need to add R Arrow to our worker nodes?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I configured AppVeyor to test Arrow related ones in R for now. So, at least it's being tested in AppVeyor.

Yes, it would be nice if we have Arrow R installed in Jenkins workers to test it out; however, I remember it's a bit tricky to install Arrow R library for now (you cannot just install via CRAN but needs some manual preparation - see https://github.com/apache/arrow/tree/master/r#installation).

I am worried if it messes up the current Jenkins worker's environment. If you see any concern, @shaneknapp, you can just don't install it for now. It's being tested via AppVeyor at least so it's neither a must or urgent.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, R arrow installed easily and w/o issue (surprising, i know!). :)

sknapp@ubuntu-testing:~$ Rscript -e "packageVersion('arrow')"
[1] ‘0.15.1.1’

i will install this on the remaining workers later today/tomorrow.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @shaneknapp !

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

}

stream_writer$write_batch(batch)
Expand Down
1 change: 1 addition & 0 deletions R/run-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
# limitations under the License.
#


FWDIR="$(cd `dirname $0`; pwd)"

FAILED=0
Expand Down
3 changes: 1 addition & 2 deletions appveyor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,8 @@ install:
- ps: .\dev\appveyor-install-dependencies.ps1
# Required package for R unit tests
- cmd: R -e "install.packages(c('knitr', 'rmarkdown', 'e1071', 'survival'), repos='https://cloud.r-project.org/')"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be better to restore this line from #26041 to install arrow?

cmd: R -e "install.packages(c('knitr', 'rmarkdown', 'e1071', 'survival', 'arrow'), repos='https://cloud.r-project.org/')"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Yes. That will be better.

# Use Arrow R 0.14.1 for now. 0.15.0 seems not working for now. See SPARK-29378.
- cmd: R -e "install.packages(c('assertthat', 'bit64', 'fs', 'purrr', 'R6', 'tidyselect'), repos='https://cloud.r-project.org/')"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line was to install Arrow dependencies manually (because devtools started to require testthat dependency as 2.0.0+; however, SparkR requires 1.0.2). So, I think we can remove this line too. Let me make a quick followup.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Go ahead!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made a followup - #26566. For clarification It actually makes no diff but only less codes.

- cmd: R -e "install.packages('https://cran.r-project.org/src/contrib/Archive/arrow/arrow_0.14.1.tar.gz', repos=NULL, type='source')"
- cmd: R -e "install.packages('https://cran.r-project.org/src/contrib/Archive/arrow/arrow_0.15.1.tar.gz', repos=NULL, type='source')"
# Here, we use the fixed version of testthat. For more details, please see SPARK-22817.
# As of devtools 2.1.0, it requires testthat higher then 2.1.1 as a dependency. SparkR test requires testthat 1.0.2.
# Therefore, we don't use devtools but installs it directly from the archive including its dependencies.
Expand Down