Fix TimestampGen to generate value not too close to the minimum allowed timestamp [databricks]

[ttnghia]

This modifies the TimestampGen class to generate values one month further from the minimum allowed value. That is because when the timestamps fall in 0001-01 month may cause issue when reading from Spark into Pyspark:

self = TimestampType, ts = -62135596800000000

    def fromInternal(self, ts):
        if ts is not None:
            # using int to avoid precision loss in float
>           return datetime.datetime.fromtimestamp(ts // 1000000).replace(microsecond=ts % 1000000)
E           ValueError: year 0 is out of range

Closes #9701.

[ttnghia]

build

[ttnghia]

build

[ttnghia]

build

[revans2]

I'm not sure that this is 100% what we want. Would it be better to do a date_diff in Spark and convert them all to Integers? And for timestamps could we use unix_micros to get the results out as longs?

@jlowe @abellina what do you think? Is it better to restrict the range of timestamps/dates and risk that we have some odd corner case where we copy it back to the host. Or is it better to try and "normalize" the data into something that python does not have issues with?

[abellina]

Reference in new is

I would do both. One case we convert to longs and keep the range, the other we don't change the query and adjust as @ttnghia has. That way we don't add extra layers that could hide an issue? My two cents.

[revans2]

I would do both. One case we convert to longs and keep the range, the other we don't change the query and adjust as @ttnghia has. That way we don't add extra layers that could hide an issue? My two cents.

I'm not sure how to do that. Converting all timestamps/dates to a different types is something that we can have an annotation for and programmatically insert it into the plan before we do a collect. I don't know now to coordinate that with datagen. But Lets check this in as is, and I'll file a follow on issue to explore if there is a better way to handle this.

[revans2]

I filed #9747

[ttnghia]

build