Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-33084][SQL][TESTS][FOLLOW-UP] Fix Scala 2.13 UT failure #30980

Closed
wants to merge 2 commits into from

Conversation

AngersZhuuuu
Copy link
Contributor

@AngersZhuuuu AngersZhuuuu commented Dec 31, 2020

What changes were proposed in this pull request?

Fix UT according to #29966 (comment)

Change StructType construct from

def inputSchema: StructType = StructType(StructField("inputColumn", LongType) :: Nil)

to

  def inputSchema: StructType = new StructType().add("inputColumn", LongType)

The whole udf class is :

package org.apache.spark.examples.sql

import org.apache.spark.sql.expressions.{MutableAggregationBuffer, UserDefinedAggregateFunction}
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row

class Spark33084 extends UserDefinedAggregateFunction {
  // Data types of input arguments of this aggregate function
  def inputSchema: StructType = new StructType().add("inputColumn", LongType)

  // Data types of values in the aggregation buffer
  def bufferSchema: StructType =
    new StructType().add("sum", LongType).add("count", LongType)
  // The data type of the returned value
  def dataType: DataType = DoubleType
  // Whether this function always returns the same output on the identical input
  def deterministic: Boolean = true
  // Initializes the given aggregation buffer. The buffer itself is a `Row` that in addition to
  // standard methods like retrieving a value at an index (e.g., get(), getBoolean()), provides
  // the opportunity to update its values. Note that arrays and maps inside the buffer are still
  // immutable.
  def initialize(buffer: MutableAggregationBuffer): Unit = {
    buffer(0) = 0L
    buffer(1) = 0L
  }
  // Updates the given aggregation buffer `buffer` with new input data from `input`
  def update(buffer: MutableAggregationBuffer, input: Row): Unit = {
    if (!input.isNullAt(0)) {
      buffer(0) = buffer.getLong(0) + input.getLong(0)
      buffer(1) = buffer.getLong(1) + 1
    }
  }
  // Merges two aggregation buffers and stores the updated buffer values back to `buffer1`
  def merge(buffer1: MutableAggregationBuffer, buffer2: Row): Unit = {
    buffer1(0) = buffer1.getLong(0) + buffer2.getLong(0)
    buffer1(1) = buffer1.getLong(1) + buffer2.getLong(1)
  }
  // Calculates the final result
  def evaluate(buffer: Row): Double = buffer.getLong(0).toDouble / buffer.getLong(1)
}

Why are the changes needed?

Fix UT for scala 2.13

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existed UT

@github-actions github-actions bot added the SQL label Dec 31, 2020
@AngersZhuuuu
Copy link
Contributor Author

FYI @dongjoon-hyun @maropu Also cc @koertkuipers

@SparkQA
Copy link

SparkQA commented Dec 31, 2020

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38149/

@SparkQA
Copy link

SparkQA commented Dec 31, 2020

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38149/

@SparkQA
Copy link

SparkQA commented Dec 31, 2020

Test build #133559 has finished for PR 30980 at commit 65b93e1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-33084][SQL][TESTS][FOLLOW-UP] Fix UT failed in scala 2.13 [SPARK-33084][SQL][TESTS][FOLLOW-UP] Fix Scala 2.13 UT failure Dec 31, 2020
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @AngersZhuuuu and @koertkuipers .
I verified this locally. Merged to master.

@maropu
Copy link
Member

maropu commented Jan 3, 2021

late LGTM. Thanks for fixing it, @AngersZhuuuu

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants