[SPARK-5726] [MLLIB] Elementwise (Hadamard) Vector Product Transformer #4580

ogeagla · 2015-02-13T02:26:08Z

See https://issues.apache.org/jira/browse/SPARK-5726

AmplabJenkins · 2015-02-13T02:27:09Z

Can one of the admins verify this patch?

mengxr · 2015-02-13T06:12:29Z

add to whitelist

mengxr · 2015-02-13T06:12:33Z

ok to test

SparkQA · 2015-02-13T06:17:30Z

Test build #27419 has started for PR 4580 at commit 9418562.

This patch merges cleanly.

SparkQA · 2015-02-13T07:31:49Z

Test build #27419 has finished for PR 4580 at commit 9418562.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-13T07:31:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27419/
Test PASSed.

SparkQA · 2015-02-13T18:27:31Z

Test build #27446 has started for PR 4580 at commit 5bd4288.

This patch merges cleanly.

mengxr · 2015-02-13T18:38:53Z

mllib/src/main/scala/org/apache/spark/ml/feature/ElementwiseProductTF.scala

+ * Maps a vector to the hadamard product of it and a reference vector.
+ */
+@AlphaComponent
+class ElementwiseProductTF extends UnaryTransformer[Vector, Vector, ElementwiseProductTF] {


What does TF mean here?

Ah, just realized the TF in HashingTF stands for 'term frequencies', not 'transformer'. Will remove. Thanks for pointing out.

SparkQA · 2015-02-13T18:57:58Z

Test build #27449 has started for PR 4580 at commit cafe8c9.

This patch merges cleanly.

SparkQA · 2015-02-13T19:01:16Z

Test build #27446 has finished for PR 4580 at commit 5bd4288.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-13T19:01:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27446/
Test FAILed.

SparkQA · 2015-02-13T19:35:23Z

Test build #27449 has finished for PR 4580 at commit cafe8c9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]
- class ElementwiseProduct(val scalingVector: Vector) extends VectorTransformer

AmplabJenkins · 2015-02-13T19:35:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27449/
Test FAILed.

ogeagla · 2015-02-13T19:57:13Z

@mengxr the unit tests and style checks for this feature pass locally, and the jenkins build actually shows the test suite also passing, but there is a compilation failure, any suggestions to get the build green?

edit: rebased on master and still occurring

the error is return exit status 1 in run-tests, this line:
| grep -v -e "info.*Resolving" -e "warn.*Merging" -e "info.*Including"

final edit: passing after another rebase

SparkQA · 2015-02-13T20:47:37Z

Test build #27453 has started for PR 4580 at commit 453a502.

This patch merges cleanly.

SparkQA · 2015-02-13T21:27:32Z

Test build #27453 has finished for PR 4580 at commit 453a502.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-13T21:27:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27453/
Test FAILed.

SparkQA · 2015-02-15T15:57:54Z

Test build #27525 has started for PR 4580 at commit a7823e5.

This patch merges cleanly.

SparkQA · 2015-02-15T17:15:53Z

Test build #27525 has finished for PR 4580 at commit a7823e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class JavaStatefulNetworkWordCount
- class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]
- class ElementwiseProduct(val scalingVector: Vector) extends VectorTransformer

AmplabJenkins · 2015-02-15T17:15:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27525/
Test PASSed.

SparkQA · 2015-03-13T22:53:03Z

Test build #28597 has started for PR 4580 at commit 7eb8f0f.

This patch merges cleanly.

SparkQA · 2015-03-14T00:10:13Z

Test build #28597 has finished for PR 4580 at commit 7eb8f0f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]
- class ElementwiseProduct(val scalingVector: Vector) extends VectorTransformer

AmplabJenkins · 2015-03-14T00:10:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28597/
Test PASSed.

SparkQA · 2015-03-26T17:28:19Z

Test build #29243 has started for PR 4580 at commit e7ff5f2.

This patch merges cleanly.

SparkQA · 2015-03-26T18:49:58Z

Test build #29243 has finished for PR 4580 at commit e7ff5f2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]
- class ElementwiseProduct(val scalingVector: Vector) extends VectorTransformer

ogeagla · 2015-05-07T18:15:19Z

Rebased.

AmplabJenkins · 2015-05-07T18:17:14Z

Merged build triggered.

AmplabJenkins · 2015-05-07T18:17:22Z

Merged build started.

SparkQA · 2015-05-07T18:17:49Z

Test build #32131 has started for PR 4580 at commit 90f7e39.

SparkQA · 2015-05-07T18:31:18Z

Test build #32131 has finished for PR 4580 at commit 90f7e39.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]
- class StringArrayParam(parent: Params, name: String, doc: String, isValid: Array[String] => Boolean)
- class ElementwiseProduct(val scalingVector: Vector) extends VectorTransformer
- class VectorAssembler(JavaTransformer, HasInputCols, HasOutputCol):
- class HasInputCols(Params):

AmplabJenkins · 2015-05-07T18:31:19Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-07T18:31:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32131/
Test FAILed.

jkbradley · 2015-05-07T18:56:53Z

@ogeagla Please look at the updated createTransformFunc method in UnaryTransformer, and change your override to match it.

AmplabJenkins · 2015-05-07T19:27:13Z

Merged build triggered.

AmplabJenkins · 2015-05-07T19:27:22Z

Merged build started.

SparkQA · 2015-05-07T19:29:04Z

Test build #32140 has started for PR 4580 at commit fac12ad.

SparkQA · 2015-05-07T21:09:17Z

Test build #32140 has finished for PR 4580 at commit fac12ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]
- class StringArrayParam(parent: Params, name: String, doc: String, isValid: Array[String] => Boolean)
- class ElementwiseProduct(val scalingVector: Vector) extends VectorTransformer
- class VectorAssembler(JavaTransformer, HasInputCols, HasOutputCol):
- class HasInputCols(Params):

AmplabJenkins · 2015-05-07T21:09:22Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-07T21:09:22Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32140/
Test PASSed.

jkbradley · 2015-05-07T21:49:31Z

LGTM Merging into master and branch-1.4 Thanks very much!

See https://issues.apache.org/jira/browse/SPARK-5726 Author: Octavian Geagla <ogeagla@gmail.com> Author: Joseph K. Bradley <joseph@databricks.com> Closes #4580 from ogeagla/spark-mllib-weighting and squashes the following commits: fac12ad [Octavian Geagla] [SPARK-5726] [MLLIB] Use new createTransformFunc. 90f7e39 [Joseph K. Bradley] small cleanups 4595165 [Octavian Geagla] [SPARK-5726] [MLLIB] Remove erroneous test case. ded3ac6 [Octavian Geagla] [SPARK-5726] [MLLIB] Pass style checks. 37d4705 [Octavian Geagla] [SPARK-5726] [MLLIB] Incorporated feedback. 1dffeee [Octavian Geagla] [SPARK-5726] [MLLIB] Pass style checks. e436896 [Octavian Geagla] [SPARK-5726] [MLLIB] Remove 'TF' from 'ElementwiseProductTF' cb520e6 [Octavian Geagla] [SPARK-5726] [MLLIB] Rename HadamardProduct to ElementwiseProduct 4922722 [Octavian Geagla] [SPARK-5726] [MLLIB] Hadamard Vector Product Transformer (cherry picked from commit 658a478) Signed-off-by: Joseph K. Bradley <joseph@databricks.com>

See https://issues.apache.org/jira/browse/SPARK-5726 Author: Octavian Geagla <ogeagla@gmail.com> Author: Joseph K. Bradley <joseph@databricks.com> Closes apache#4580 from ogeagla/spark-mllib-weighting and squashes the following commits: fac12ad [Octavian Geagla] [SPARK-5726] [MLLIB] Use new createTransformFunc. 90f7e39 [Joseph K. Bradley] small cleanups 4595165 [Octavian Geagla] [SPARK-5726] [MLLIB] Remove erroneous test case. ded3ac6 [Octavian Geagla] [SPARK-5726] [MLLIB] Pass style checks. 37d4705 [Octavian Geagla] [SPARK-5726] [MLLIB] Incorporated feedback. 1dffeee [Octavian Geagla] [SPARK-5726] [MLLIB] Pass style checks. e436896 [Octavian Geagla] [SPARK-5726] [MLLIB] Remove 'TF' from 'ElementwiseProductTF' cb520e6 [Octavian Geagla] [SPARK-5726] [MLLIB] Rename HadamardProduct to ElementwiseProduct 4922722 [Octavian Geagla] [SPARK-5726] [MLLIB] Hadamard Vector Product Transformer

ogeagla changed the title ~~[SPARK-5726] [MLLIB] Hadamard Vector Product Transformer~~ [SPARK-5726] [MLLIB] Elementwise (Hadamard) Vector Product Transformer Feb 13, 2015

mengxr reviewed Feb 13, 2015
View reviewed changes

ogeagla force-pushed the spark-mllib-weighting branch from cafe8c9 to 453a502 Compare February 13, 2015 20:43

ogeagla force-pushed the spark-mllib-weighting branch from 453a502 to a7823e5 Compare February 15, 2015 15:55

ogeagla force-pushed the spark-mllib-weighting branch from a7823e5 to 7eb8f0f Compare March 13, 2015 22:47

ogeagla force-pushed the spark-mllib-weighting branch from 7eb8f0f to e7ff5f2 Compare March 26, 2015 17:24

ogeagla and others added 8 commits May 7, 2015 12:14

[SPARK-5726] [MLLIB] Hadamard Vector Product Transformer

4922722

[SPARK-5726] [MLLIB] Rename HadamardProduct to ElementwiseProduct

cb520e6

[SPARK-5726] [MLLIB] Remove 'TF' from 'ElementwiseProductTF'

e436896

[SPARK-5726] [MLLIB] Pass style checks.

1dffeee

[SPARK-5726] [MLLIB] Incorporated feedback.

37d4705

[SPARK-5726] [MLLIB] Pass style checks.

ded3ac6

[SPARK-5726] [MLLIB] Remove erroneous test case.

4595165

small cleanups

90f7e39

ogeagla force-pushed the spark-mllib-weighting branch from 3ea9b71 to 90f7e39 Compare May 7, 2015 18:15

[SPARK-5726] [MLLIB] Use new createTransformFunc.

fac12ad

asfgit closed this in 658a478 May 7, 2015

[SPARK-5726] [MLLIB] Elementwise (Hadamard) Vector Product Transformer #4580

[SPARK-5726] [MLLIB] Elementwise (Hadamard) Vector Product Transformer #4580

Conversation

ogeagla commented Feb 13, 2015

AmplabJenkins commented Feb 13, 2015

mengxr commented Feb 13, 2015

mengxr commented Feb 13, 2015

SparkQA commented Feb 13, 2015

SparkQA commented Feb 13, 2015

AmplabJenkins commented Feb 13, 2015

SparkQA commented Feb 13, 2015

mengxr Feb 13, 2015

Choose a reason for hiding this comment

ogeagla Feb 13, 2015

Choose a reason for hiding this comment

SparkQA commented Feb 13, 2015

SparkQA commented Feb 13, 2015

AmplabJenkins commented Feb 13, 2015

SparkQA commented Feb 13, 2015

AmplabJenkins commented Feb 13, 2015

ogeagla commented Feb 13, 2015

SparkQA commented Feb 13, 2015

SparkQA commented Feb 13, 2015

AmplabJenkins commented Feb 13, 2015

SparkQA commented Feb 15, 2015

SparkQA commented Feb 15, 2015

AmplabJenkins commented Feb 15, 2015

SparkQA commented Mar 13, 2015

SparkQA commented Mar 14, 2015

AmplabJenkins commented Mar 14, 2015

SparkQA commented Mar 26, 2015

SparkQA commented Mar 26, 2015

ogeagla commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

jkbradley commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

jkbradley commented May 7, 2015