Allow static credentials for all AWS clients, not only for S3

[morozov]

Feature Request / Improvement

I want to register an Iceberg catalog of type AWS Glue in Flink with the code like this:

var properties = new HashMap<>();
properties.put("type", "iceberg");
properties.put("catalog-type", "glue");
properties.put("catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog");
properties.put("warehouse", "s3://my-bucket/path/to/warehouse");

var factory = new FlinkCatalogFactory();
var catalog = factory.createCatalog("glue_catalog", properties);

For authentication with AWS, I want to use static client credentials (a pair of access key ID and secret). I couldn't find the user-facing documentation on how do do that, so I resorted to reading the source code.

Currently, static client credentials can be only configured for the S3 client:

iceberg/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java

Lines 192 to 216 in 0e7aa84

	/**
	* Configure the static access key ID used to access S3FileIO.
	*
	* <p>When set, the default client factory will use the basic or session credentials provided
	* instead of reading the default credential chain to create S3 access credentials. If {@link
	* #SESSION_TOKEN} is set, session credential is used, otherwise basic credential is used.
	*/
	public static final String ACCESS_KEY_ID = "s3.access-key-id";
	/**
	* Configure the static secret access key used to access S3FileIO.
	*
	* <p>When set, the default client factory will use the basic or session credentials provided
	* instead of reading the default credential chain to create S3 access credentials. If {@link
	* #SESSION_TOKEN} is set, session credential is used, otherwise basic credential is used.
	*/
	public static final String SECRET_ACCESS_KEY = "s3.secret-access-key";
	/**
	* Configure the static session token used to access S3FileIO.
	*
	* <p>When set, the default client factory will use the session credentials provided instead of
	* reading the default credential chain to create S3 access credentials.
	*/
	public static final String SESSION_TOKEN = "s3.session-token";

These credentials do not affect the other clients, e.g. the AWS Glue one. As a result, in order to configure static credentials for the entire Glue catalog, one needs to implement a custom credentials provider using these parameters:

iceberg/aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java

Lines 37 to 60 in c68abfc

	/**
	* Configure the AWS credentials provider used to create AWS clients. A fully qualified concrete
	* class with package that implements the {@link AwsCredentialsProvider} interface is required.
	*
	* <p>Additionally, the implementation class must also have a create() or create(Map) method
	* implemented, which returns an instance of the class that provides aws credentials provider.
	*
	* <p>Example:
	* client.credentials-provider=software.amazon.awssdk.auth.credentials.SystemPropertyCredentialsProvider
	*
	* <p>When set, the default client factory {@link
	* org.apache.iceberg.aws.AwsClientFactories#defaultFactory()} and other AWS client factory
	* classes will use this provider to get AWS credentials provided instead of reading the default
	* credential chain to get AWS access credentials.
	*/
	public static final String CLIENT_CREDENTIALS_PROVIDER = "client.credentials-provider";
	/**
	* Used by the client.credentials-provider configured value that will be used by {@link
	* org.apache.iceberg.aws.AwsClientFactories#defaultFactory()} and other AWS client factory
	* classes to pass provider-specific properties. Each property consists of a key name and an
	* associated value.
	*/
	protected static final String CLIENT_CREDENTIAL_PROVIDER_PREFIX = "client.credentials-provider.";

For example:

iceberg/aws/src/test/java/org/apache/iceberg/aws/TestAwsClientFactories.java

Lines 243 to 253 in 7071dc1

	private static class DummyValidProvider implements AwsCredentialsProvider {
	public static DummyValidProvider create() {
	return new DummyValidProvider();
	}
	@Override
	public AwsCredentials resolveCredentials() {
	return AwsBasicCredentials.create("test-accessKeyId", "test-secretAccessKey");
	}
	}

Would a PR be accepted that in addition to the s3.* parameters mentioned above added the support for similarly named aws.* parameters? Such parameters would consistently apply to all AWS clients instantiated by DefaultAwsClientFactoryand could eventually deprecate the s3.* ones..

Query engine

None

[nastra]

@jackye1995 could you (or anyone on your team) take a look at this one please?