Move the database configuration to a new section

[mik-laj]

Hello.

Currently, the core section has 48 configuration options. This is a bit of a lot and it becomes problematic to find the configuration option that interests us. Our users ask questions about an option that already exists by this.

Below I present the statistics, how many options each section has.

$ yq eval . --tojson airflow/config_templates/config.yml | jq -r 'map({key:.name, value:(.options|length)})|sort_by(.value)|reverse |.[]| [.key, .value]|@tsv'
core	48
webserver	46
scheduler	24
celery	24
kubernetes	21
logging	20
smtp	9
api	9
metrics	9
elasticsearch	7
operators	7
email	6
kerberos	5
atlas	5
smart_sensor	4
dask	4
elasticsearch_configs	2
sentry	2
hive	2
cli	2
secrets	2
github_enterprise	1
celery_broker_transport_options	1
celery_kubernetes_executor	1
lineage	1
debug	1

It would be great if we could move some options to the new section. I think we can start with the database-related options that we can move to the new [database] section.

sql_alchemy_conn
sql_engine_encoding
sql_engine_collation_for_ids
sql_alchemy_pool_enabled
sql_alchemy_pool_size
sql_alchemy_max_overflow
sql_alchemy_pool_recycle
sql_alchemy_pool_pre_ping
sql_alchemy_schema
sql_alchemy_connect_args
load_default_connections
max_db_retries

We've made similar changes in the past.
#12165
#6887

I also wonder what other sections we can divide to make the configuration even easier. What do you think that the configuration options for our flask application should be moved to the webui section? This is to separate the infrastructure/webserver from the UI/browser options.

[junnplus]

Hi @mik-laj, can I work on this ?

[mik-laj]

Assigned.

[kelechi2020]

@potiuk @junnplus
Let me finish this up.
#15945 (comment)