[BUG] Loading columns from an ORC file without column names returns no data

[razajafri]

commit b745041f698120be21ab889706880e976a599fdb
Author: SaurabhChawla saurabhc@qubole.com
Date: Thu Jul 16 13:11:47 2020 +0000

[SPARK-32234][SQL] Spark sql commands are failing on selecting the orc tables

[kuhushukla]

@razajafri Can u elaborate on what the fix would be and change the priority if needed?

[razajafri]

     val table = """CREATE TABLE `test_orc_data` (
      `_col1` INT,
      `_col2` STRING,
      `_col3` INT)
      USING orc"""
    
    spark.sql(table).collect
    
    spark.sql("insert into test_orc_data values(13, '155', 2020)").collect
    
    val df = """select _col2 from test_orc_data limit 5"""
    spark.sql(df).collect

If we run the above example in 3.0.1 with rapids-plugin turned off, it results in the following

res7: Array[org.apache.spark.sql.Row] = Array([155])

With the rapids-plugin turned on it results in

20/11/11 00:25:05 WARN SchemaEvolution: Column names are missing from this file. This is caused by a writer earlier than HIVE-4243. The reader will reconcile schemas based on index. File type: struct<_col1:int,_col2:string,_col3:int>, reader type: struct<_col2:string>
res3: Array[org.apache.spark.sql.Row] = Array([])

It seems like the GPU version doesn't crash but reports an incorrect results because the column names aren't passed