We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
I wish we can support GetMapValue with non-literal key.
For example:
from pyspark.sql.types import * from pyspark.sql.functions import * jsonString="""{"Zipcode":704,"ZipCodeType":"STANDARD","City":"PARC PARQUE","State":"PR","Zipcode":999}""" df=spark.createDataFrame([(1, jsonString)],["id","value"]) df2=df.withColumn("value",from_json(df.value,MapType(StringType(),StringType()))) df2.withColumn('newcol', expr(f""" map_filter(value, (k,v) -> value[split(k, "abc")[0]] is not NULL ) """) ).collect()
Not-supported messages:
!Expression <GetMapValue> value#136[split(lambda k#225, abc, 2)[0]] cannot run on GPU because key only supports StringType if it is a literal value
The text was updated successfully, but these errors were encountered:
@mythrocks is this a duplicate of #5204 ?
Sorry, something went wrong.
Yep, it's a dupe. The libcudf end of it supports only scalar keys, currently.
libcudf
scalar
It's not hard to add support for vectors. Just needs doing.
No branches or pull requests
I wish we can support GetMapValue with non-literal key.
For example:
Not-supported messages:
The text was updated successfully, but these errors were encountered: