[FEA] Binary type encoding for Parquet writes

[jlowe]

Is your feature request related to a problem? Please describe.
Apache Spark supports writing binary data to Parquet as Parquet's binary (BYTE_ARRAY) type. The RAPIDS Accelerator needs to emulate this behavior for binary data in order to accelerate Parquet writes containing binary types.

Describe the solution you'd like
The cuio column_metadata already contains a commented-out field that would indicate whether the data should be encoded as a binary type. Incoming data that is either a string or a LIST of INT8/UINT8 would be encoded as Parquet binary if this flag is set.

Describe alternatives you've considered
cuio could infer that a column that is a LIST of INT8/UINT8 should be written as binary, but there are some use-cases that would break with that assumption. For example, if someone used a Spark type of ArrayType(ByteType) then it would appear in cudf as LIST of INT8, but that should not be binary-encoded.

[devavret]

I cannot find in the parquet spec, how to distinguish between string and binary type when the physical type is BYTE_ARRAY. There is no binary type in the LogicalType or ConvertedType structs.

This means you can just encode your current data as a binary type and pass it as a string column. You'd have to pass this type information to the reader on your own.

Do you have any parquet files I can look at that contain binary data.

[jlowe]

Here's a sample Parquet file for reference, written by Apache Spark, that contains a string column, a list-of-bytes column, and a binary column.

string-list-binary.parquet.zip

It's clear from the metadata that Spark is encoding extra metadata information to distinguish strings from raw binary types, but otherwise from the raw Parquet metadata they appear to be identical. Here's the output of the metadata from Parquet's CLI tool:

File path:  /tmp/string-list-binary.parquet
Created by: parquet-mr version 1.12.2 (build 77e30c8093386ec52c3cfa6c34b7ef3321322c94)
Properties:
                   org.apache.spark.version: 3.2.1
  org.apache.spark.sql.parquet.row.metadata: {"type":"struct","fields":[{"name":"string","type":"string","nullable":true,"metadata":{}},{"name":"list","type":{"type":"array","elementType":"byte","containsNull":true},"nullable":true,"metadata":{}},{"name":"binary","type":"binary","nullable":true,"metadata":{}}]}
Schema:
message spark_schema {
  optional binary string (STRING);
  optional group list (LIST) {
    repeated group list {
      optional int32 element (INTEGER(8,true));
    }
  }
  optional binary binary;
}


Row group 0:  count: 1  125.00 B records  start: 4  total: 125 B
--------------------------------------------------------------------------------
                   type      encodings count     avg size   nulls   min / max
string             BINARY    S   _     1         38.00 B    0       "123" / "123"
list.list.element  INT32     S   _     3         16.33 B    0       "1" / "3"
binary             BINARY    S   _     1         38.00 B    0       "0x010203" / "0x010203"

I agree this is not a feature requirement of the libcudf writer, and instead is more of a requirement of the high-level reader code to reinterpret string data as binary data (or vice-versa) based on what schema is expected.

[revans2]

In the SchemaElement the LogicalType and the ConvertedType are both optional. For a BINARY the Type is BYTE_ARRAY and neither LogicalType not ConvertedType are set.

https://github.com/apache/parquet-mr/blob/a2da156b251d13bce1fa81eb95b555da04880bc1/parquet-column/src/main/java/org/apache/parquet/schema/PrimitiveType.java#L608-L625

Prints out the type for the Schema in the output above. If there is a LogicalTypeAnnotation, The java APIs combined LogicalType and ConvertedType, then it is printed out in parentheses. This happens for optional binary string (STRING);, but not for optional binary binary;

This is the difference. If we write the binary columns out as strings they will have the StringType LogicalType and the UTF8 ConvertedType. Not sure what happens with Spark if it sees this mismatch between the schema it wrote out and the one in parquet. It will probably just auto cast it back to binary, but for anything that does not read the Spark metadata schema they might interpret it incorrectly.

Technically for reads we should be looking at this metadata too and only return a DType.STRING if UTF8/StringType is set, but older versions of parquet didn't use that so we need a flag to say if you care about it or not.

[revans2]

Actually I need to correct it a bit, at least for what Spark does on reads.

BINARY with an annotation of STRING, ENUM, UUID, and JSON should also be returned as DType.STRING.
BINARY with no LogicalAnnotation or BSON are returned as Spark's binary type.

[devavret]

Ok so you want parquet writer to not write any converted type or logical type when you pass the column option

cudf/cpp/include/cudf/io/types.hpp

Line 252 in 6acf226

// bool _output_as_binary = false;

.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[sameerz]

Still needed.

[hyperbolic2346]

fixed by #11328 and #11160