Alter table to generic types

[fcomuniz]

Hi, this project is very awesome, we are starting to use it now for saving our data using iceberg and it is working very well.

One issue we have encountered is when we want to change the data type from an int to a string, just so we can have a more generic data type to store the given information. This happens when there is a schema change from the source and we don't want to lose the data that was in that column.

For example, from spark-sql

create table iceberg.bronze.test__test (id int) using iceberg;
alter table iceberg.bronze.test__test alter column id type string;

Gives the following error

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalArgumentException: Cannot change column type: id: long -> string
        at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:459)
        at org.apache.iceberg.SchemaUpdate.updateColumn(SchemaUpdate.java:244)
        at org.apache.iceberg.spark.Spark3Util.applySchemaChanges(Spark3Util.java:160)
        at org.apache.iceberg.spark.SparkCatalog.commitChanges(SparkCatalog.java:432)
        at org.apache.iceberg.spark.SparkCatalog.alterTable(SparkCatalog.java:216)
        at org.apache.iceberg.spark.SparkCatalog.alterTable(SparkCatalog.java:79)
        at org.apache.spark.sql.execution.datasources.v2.AlterTableExec.run(AlterTableExec.scala:37)
        ... 44 more

I would just like some pointers as to how i could change the data type of the data, and an idea as to how it could be implemented into the alter table statement.

Thanks for any help you can give.

[paulboocock]

We have two common type evolutions that we'd like to see:

int -> float
int -> string

Actually, for the latter it's more like:
numeric -> string

This is for similar reasons as the initial comment, where the source data starts as one type but as it changes we don't want to lose that historical data

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.