[BUG] to_date with two digit year uses different century between CPU and GPU

[andygrove]

Describe the bug
I enabled spark.rapids.sql.incompatibleDateFormats.enabled and attempted to parse 31/12/99 using the format dd/MM/yy.

Spark on CPU returns 2099-12-31 but Spark on GPU returns 1999-12-31.

Steps/Code to reproduce bug

Update ParseDateTimeSuite.scala:

• Add value "31/12/99" to timestampValues variable

• Add new test:

testSparkResultsAreEqual("to_date dd/MM/yy",
  datesAsStrings,
  conf = new SparkConf().set(SQLConf.LEGACY_TIME_PARSER_POLICY.key, "CORRECTED")) {
  df => df.withColumn("c1", to_date(col("c0"), "dd/MM/yy"))
}

Expected behavior
Test should pass or we should fall back to CPU for yy.

Additional context
Spark 3.1.1

[andygrove]

I plan on disabling yy on GPU for now and filing a follow-on issue to handle this consistently with Spark.

cuDF parsing is based on strptime and I found this in the strptime docs, so I am assuming for now that cuDF does the same but I need to confirm this as well as investigate the rules that Spark implements.

The year within century. When a century is not otherwise specified, 
values in the range [69,99] shall refer to years 1969 to 1999 inclusive, and 
values in the range [00,68] shall refer to years 2000 to 2068 inclusive;
leading zeros shall be permitted but shall not be required.

Spark uses DateTimeFormatter which has different rules:

For parsing, this will parse using the base value of 2000, resulting in a year within the range 2000 to 2099 inclusive.

[sameerz]

We will now fall back to the CPU but leaving this open so we can fix the correctness issue on the GPU in the longer run.