Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Support last in windowing context for DoubleType #6193

Open
res-life opened this issue Aug 2, 2022 · 0 comments
Open

[FEA] Support last in windowing context for DoubleType #6193

res-life opened this issue Aug 2, 2022 · 0 comments
Labels
feature request New feature or request

Comments

@res-life
Copy link
Collaborator

res-life commented Aug 2, 2022

I wish we can support function last in windowing context for IntegerType/DoubleType.

For example:

import org.apache.spark.sql.Row
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._

val data = Seq(
    Row(Row("Adam ","","Green"),"1","M",1000.1, "2019-01-01",List("Java","Scala")),
    Row(Row("Bob ","Middle","Green"),"2","M",2000.2, "2019-01-02",List("Java","Python")),
    Row(Row("Cathy ","","Green"),"3","F",3000.3, "2019-01-03",List())
)

val schema = (new StructType()
  .add("name",new StructType()
    .add("firstname",StringType)
    .add("middlename",StringType)
    .add("lastname",StringType)) 
  .add("id",StringType)
  .add("gender",StringType)
  .add("salary",DoubleType)
  .add("birthdayStr",StringType)
  .add("language",ArrayType(StringType))
             )

val df = spark.createDataFrame(spark.sparkContext.parallelize(data),schema)
df.withColumn("birthday", to_date(col("birthdayStr"))).write.format("parquet").mode("overwrite").save("/tmp/testparquet")
val df2 = spark.read.parquet("/tmp/testparquet")
df2.createOrReplaceTempView("df2")
df2.printSchema

sql("""SELECT gender, last(salary,true) OVER (PARTITION BY gender ORDER BY salary) FROM df2""").collect

Not-Supported messages:

!Expression <Last> last(salary#555) ignore nulls cannot run on GPU because input expression AttributeReference salary#555 (DoubleType is not supported); expression Last last(salary#555) ignore nulls produces an unsupported type DoubleType
@res-life res-life added feature request New feature or request ? - Needs Triage Need team to review and classify labels Aug 2, 2022
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Aug 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants