[SPARK-38314][SQL] Fix of failing to read parquet files after writing the hidden file metadata in

[Yaohua628]

What changes were proposed in this pull request?

Selecting and then writing df containing hidden file metadata column _metadata into a file format like parquet, delta will still keep the internal Attribute metadata information. Then when reading those parquet, delta files again, it will actually break the code, because it wrongly thinks user data schema_metadata is a hidden file source metadata column.

// prepare a file source df
df.select("*", "_metadata").write.format("parquet").save(path)

spark.read.format("parquet").load(path).select("*").show()

This PR fixes this by cleaning up any remaining metadata information of output columns.

Why are the changes needed?

Bugfix

Does this PR introduce any user-facing change?

No

How was this patch tested?

A new UT

[Yaohua628]

@cloud-fan please take a look whenever you have a chance, thanks a lot, appreciate the help!

[AmplabJenkins]

Can one of the admins verify this patch?

[Yaohua628]

@cloud-fan hi Wenchen, are we ready to merge this fix? thanks!

[cloud-fan]

thanks, merging to master!