Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make map column non-nullable when it's a key in another map. #9147

Merged
merged 3 commits into from
Sep 7, 2023
Merged

Make map column non-nullable when it's a key in another map. #9147

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
65d90f9
Make map column non-nullable
Aug 31, 2023
aba56d3
Update
Sep 4, 2023
b08edb9
Merge branch 'branch-23.10' into fix-map-nullable
Sep 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions sql-plugin/src/main/scala/com/nvidia/spark/rapids/SchemaUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ import org.apache.spark.sql.types._

object SchemaUtils {
// Parquet field ID metadata key
val FIELD_ID_METADATA_KEY = "parquet.field.id"
private val FIELD_ID_METADATA_KEY = "parquet.field.id"

/**
* Convert a TypeDescription to a Catalyst StructType.
Expand Down Expand Up @@ -283,6 +283,7 @@ object SchemaUtils {
builder.withMapColumn(
mapColumn(name,
writerOptionsFromField(
// This nullable is useless because we use the child of struct column
structBuilder(name, nullable),
m.keyType,
"key",
Expand All @@ -295,7 +296,12 @@ object SchemaUtils {
m.valueContainsNull,
writeInt96,
fieldMeta,
parquetFieldIdWriteEnabled).build().getChildColumnOptions()(0)))
parquetFieldIdWriteEnabled).build().getChildColumnOptions()(0),
// set the nullable for this map
// if `m` is a key of another map, this `nullable` should be false
// e.g.: map1(map2(int,int), int), the map2 is the map
// key of map1, map2 should be non-nullable
nullable))
case BinaryType =>
if (parquetFieldIdWriteEnabled && parquetFieldId.nonEmpty) {
builder.withBinaryColumn(name, nullable, parquetFieldId.get)
Expand Down