SPARK-1782: svd for sparse matrix using ARPACK #964

vrilleup · 2014-06-04T01:23:57Z

copy ARPACK dsaupd/dseupd code from latest breeze
change RowMatrix to use sparse SVD
change tests for sparse SVD

All tests passed. I will run it against some large matrices.

copy ARPACK dsaupd/dseupd code from latest breeze change RowMatrix to use sparse SVD change tests for sparse SVD

AmplabJenkins · 2014-06-04T01:27:59Z

Can one of the admins verify this patch?

mateiz · 2014-06-04T06:06:50Z

Jenkins, this is ok to test

AmplabJenkins · 2014-06-04T06:07:59Z

Merged build triggered.

AmplabJenkins · 2014-06-04T06:08:08Z

Merged build started.

AmplabJenkins · 2014-06-04T06:09:40Z

Merged build finished.

AmplabJenkins · 2014-06-04T06:09:41Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15436/

AmplabJenkins · 2014-06-04T07:42:58Z

Merged build triggered.

AmplabJenkins · 2014-06-04T07:43:04Z

Merged build started.

AmplabJenkins · 2014-06-04T08:21:24Z

Merged build finished.

AmplabJenkins · 2014-06-04T08:21:25Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15438/

mengxr · 2014-06-04T08:24:52Z

@vrilleup It didn't pass the tests. Please check the Jenkins page.

mengxr · 2014-06-04T08:27:02Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala

+   */
+  private[mllib] def multiplyGramianMatrix(v: Vector): Vector = {
+    val bv = rows.map{
+      row => row.toBreeze * row.toBreeze.dot(v.toBreeze)


This would create many temp objects. Please use RDD#aggregate and axpy. The initial vector should be a dense vector.

You can use the code in computeGramianMatrix as an example.

changed to use aggregrate and axpy (from breeze). There are still some boxing/unboxing overhead from/to breeze objects. I can further improve if needed.

AmplabJenkins · 2014-06-04T08:27:58Z

Merged build triggered.

AmplabJenkins · 2014-06-04T08:28:06Z

Merged build started.

mengxr · 2014-06-04T08:28:20Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala

+   *
+   * The decomposition is computed by providing a function that multiples a vector with A'A to
+   * ARPACK, and iteratively invoking ARPACK-dsaupd on master node, from which we recover S and V.
+   * Then we compute U via easy matrix multiplication as U =  A * (V * S-1).


"S-1" -> "S^-1" or "S^{-1}"

fixed. Somehow my intellij treated ^ as a tag and matched it with a ^ in another line. They both got deleted when I was editing the other line...

mengxr · 2014-06-04T08:31:40Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala

@@ -494,4 +519,4 @@ object RowMatrix {

    Matrices.dense(n, n, G.data)
  }
-}
+}


add a newline at the end.

AmplabJenkins · 2014-06-04T08:32:58Z

Merged build triggered.

AmplabJenkins · 2014-06-04T08:33:06Z

Merged build started.

mengxr · 2014-06-04T08:39:31Z

mllib/src/main/scala/org/apache/spark/mllib/linalg/EigenValueDecomposition.scala

+   * @return a dense vector of eigenvalues in descending order and a dense matrix of eigenvectors
+   *         (columns of the matrix). The number of computed eigenvalues might be smaller than k.
+   */
+  private[mllib] def symmetricEigs(mul: Vector => Vector, n: Int, k: Int, tol: Double)


Is it implemented in breeze-0.8? If so, please add a TODO here so we will switch to the breeze implementation if we upgrade breeze.

current implementation in breeze-0.8 is for svd, it computes both U and V in memory. I can refactor that part to move the code block calling ARPACK to eig so only V is computed. TODO added

AmplabJenkins · 2014-06-04T09:08:31Z

Merged build finished.

AmplabJenkins · 2014-06-04T09:08:31Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15440/

AmplabJenkins · 2014-06-04T09:14:15Z

Merged build finished.

AmplabJenkins · 2014-06-04T09:14:15Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15441/

AmplabJenkins · 2014-06-04T23:57:58Z

Merged build triggered.

AmplabJenkins · 2014-07-09T07:51:19Z

Merged build finished.

AmplabJenkins · 2014-07-09T07:51:19Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16449/

AmplabJenkins · 2014-07-09T17:31:22Z

Merged build triggered.

AmplabJenkins · 2014-07-09T17:31:28Z

Merged build started.

vrilleup · 2014-07-09T18:00:30Z

@mengxr I tested axpy, there is no visible change in running time. Should be good to merge :)

mengxr · 2014-07-09T18:08:36Z

Jenkins, retest this please.

mengxr · 2014-07-09T18:11:09Z

@vrilleup Great! I'll merge it if Jenkins is happy. Thank you for the contribution!

AmplabJenkins · 2014-07-09T18:11:12Z

Merged build triggered.

AmplabJenkins · 2014-07-09T18:11:19Z

Merged build started.

AmplabJenkins · 2014-07-09T18:13:08Z

Merged build finished.

AmplabJenkins · 2014-07-09T18:13:08Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16470/

mengxr · 2014-07-09T18:46:54Z

Jenkins, retest this please.

AmplabJenkins · 2014-07-09T18:51:16Z

Merged build triggered.

AmplabJenkins · 2014-07-09T18:51:23Z

Merged build started.

AmplabJenkins · 2014-07-09T18:58:30Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-07-09T18:58:31Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16471/

rezazadeh · 2014-07-09T19:13:56Z

Thanks @vrilleup !

mengxr · 2014-07-09T19:15:43Z

Merged. Thanks!

vrilleup · 2014-07-09T19:30:22Z

@mengxr @rezazadeh Thank you! Excited to see this merged :)

AmplabJenkins · 2014-07-09T19:35:07Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-07-09T19:35:08Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16473/

vrilleup · 2014-07-09T23:16:11Z

@mengxr I ran one of the large size jobs again, per multiplication time increased to 8 seconds from 2 seconds (the version before yesterday). I don't know which part was the cause (shouldn't be the axpy part because I already compared this). Will submit a patch after identifying the cause.

copy ARPACK dsaupd/dseupd code from latest breeze change RowMatrix to use sparse SVD change tests for sparse SVD All tests passed. I will run it against some large matrices. Author: Li Pu <lpu@twitter.com> Author: Xiangrui Meng <meng@databricks.com> Author: Li Pu <li.pu@outlook.com> Closes apache#964 from vrilleup/master and squashes the following commits: 7312ec1 [Li Pu] very minor comment fix 4c618e9 [Li Pu] Merge pull request #1 from mengxr/vrilleup-master a461082 [Xiangrui Meng] make superscript show up correctly in doc 861ec48 [Xiangrui Meng] simplify axpy 62969fa [Xiangrui Meng] use BDV directly in symmetricEigs change the computation mode to local-svd, local-eigs, and dist-eigs update tests and docs c273771 [Li Pu] automatically determine SVD compute mode and parameters 7148426 [Li Pu] improve RowMatrix multiply 5543cce [Li Pu] improve svd api 819824b [Li Pu] add flag for dense svd or sparse svd eb15100 [Li Pu] fix binary compatibility 4c7aec3 [Li Pu] improve comments e7850ed [Li Pu] use aggregate and axpy 827411b [Li Pu] fix EOF new line 9c80515 [Li Pu] use non-sparse implementation when k = n fe983b0 [Li Pu] improve scala style 96d2ecb [Li Pu] improve eigenvalue sorting e1db950 [Li Pu] SPARK-1782: svd for sparse matrix using ARPACK

copy ARPACK dsaupd/dseupd code from latest breeze change RowMatrix to use sparse SVD change tests for sparse SVD All tests passed. I will run it against some large matrices. Author: Li Pu <lpu@twitter.com> Author: Xiangrui Meng <meng@databricks.com> Author: Li Pu <li.pu@outlook.com> Closes apache#964 from vrilleup/master and squashes the following commits: 7312ec1 [Li Pu] very minor comment fix 4c618e9 [Li Pu] Merge pull request apache#1 from mengxr/vrilleup-master a461082 [Xiangrui Meng] make superscript show up correctly in doc 861ec48 [Xiangrui Meng] simplify axpy 62969fa [Xiangrui Meng] use BDV directly in symmetricEigs change the computation mode to local-svd, local-eigs, and dist-eigs update tests and docs c273771 [Li Pu] automatically determine SVD compute mode and parameters 7148426 [Li Pu] improve RowMatrix multiply 5543cce [Li Pu] improve svd api 819824b [Li Pu] add flag for dense svd or sparse svd eb15100 [Li Pu] fix binary compatibility 4c7aec3 [Li Pu] improve comments e7850ed [Li Pu] use aggregate and axpy 827411b [Li Pu] fix EOF new line 9c80515 [Li Pu] use non-sparse implementation when k = n fe983b0 [Li Pu] improve scala style 96d2ecb [Li Pu] improve eigenvalue sorting e1db950 [Li Pu] SPARK-1782: svd for sparse matrix using ARPACK

SPARK-1782: svd for sparse matrix using ARPACK

e1db950

copy ARPACK dsaupd/dseupd code from latest breeze change RowMatrix to use sparse SVD change tests for sparse SVD

improve eigenvalue sorting

96d2ecb

improve scala style

fe983b0

use non-sparse implementation when k = n

9c80515

mengxr reviewed Jun 4, 2014
View reviewed changes

fix EOF new line

827411b

mengxr reviewed Jun 4, 2014
View reviewed changes

use aggregate and axpy

e7850ed

very minor comment fix

7312ec1

asfgit closed this in 1f33e1f Jul 9, 2014

SPARK-1782: svd for sparse matrix using ARPACK #964

SPARK-1782: svd for sparse matrix using ARPACK #964

Conversation

vrilleup commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

mateiz commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

mengxr commented Jun 4, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jun 4, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

vrilleup commented Jul 9, 2014

mengxr commented Jul 9, 2014

mengxr commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

mengxr commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

rezazadeh commented Jul 9, 2014

mengxr commented Jul 9, 2014

vrilleup commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

AmplabJenkins commented Jul 9, 2014

vrilleup commented Jul 9, 2014