[FEA] Add maven profiles for testing with AQE on or off

[andygrove]

Is your feature request related to a problem? Please describe.
We now have different code paths depending on whether AQE is enabled or not. We also support AQE with three versions of spark. We are currently not testing AQE very extensively (we just enable it on the integration tests) and this means regressions can slip in.

Describe the solution you'd like
I would like to add maven profiles to run the full Scala test suite either with AQE on or off, similar to how we use maven profiles to select the Spark version to test with.

Describe alternatives you've considered
We could run all the tests both with AQE on and off but this would take a long time to run. It would be better to have separate parallel builds in CI for each configuration we want to test.

Additional context
N/A

[revans2]

I am a little concerned about the impact this will have to the runtime of the tests. We already have to support Apache Spark 3.0.0, 3.0.1, and 3.1.0. We are also starting to support Databricks runtime 7.0, 7.1 and 7.2. I would much rather have us be more surgical in the tests. I'd like to see us have specific tests around the parts of AQE that we want to support, like skewed join. I'd also like to see us run some of the complex queries, like mortgage, tpch/tpcds etc with this on because it is designed for these types of queries.

[andygrove]

Ok. We do already run mortgage and tpch integrationt tests both with AQE on and off. I will look into writing specific tests for skewed join as the next step.