Add support for dynamic application configurations

[tianleh]

Description

The PR is to add application configuration service. See related issue for context. The code is mostly extracted from my other PR #5641 Once the current PR is merged, I will update the related PR to use the function provided by current PR.

We plan to release this feature in 2.13.0 and may need a backport to 2.x branch.

Note to the maintainers: please help add the backport label for 2.x

Issues Resolved

closes #5796

Screenshot

Testing the changes

Check List

• All tests pass
  • yarn test:jest
  • yarn test:jest_integration
• New functionality includes testing.
• New functionality has been documented.
• Update CHANGELOG.md
• Commits are signed per the DCO using --signoff

[codecov]

Codecov Report

Attention: Patch coverage is 87.50000% with 9 lines in your changes are missing coverage. Please review.

Project coverage is 67.03%. Comparing base (6d5560e) to head (289cab1).
Report is 9 commits behind head on main.

❗ Current head 289cab1 differs from pull request most recent head 4d80ad6. Consider uploading reports for the commit 4d80ad6 to get more accurate results

Files	Patch %	Lines
.../plugins/application_config/server/routes/index.ts	76.47%	8 Missing ⚠️
...lication_config/server/opensearch_config_client.ts	97.36%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5855      +/-   ##
==========================================
+ Coverage   67.01%   67.03%   +0.01%     
==========================================
  Files        3310     3309       -1     
  Lines       63647    63686      +39     
  Branches    10165    10167       +2     
==========================================
+ Hits        42656    42689      +33     
- Misses      18522    18526       +4     
- Partials     2469     2471       +2     

Flag	Coverage Δ
Linux_1	35.21% <ø> (ø)
Linux_2	55.09% <ø> (ø)
Linux_3	43.73% <87.50%> (-0.65%)	⬇️
Linux_4	35.20% <ø> (ø)
Windows_1	35.24% <ø> (ø)
Windows_2	55.06% <ø> (ø)
Windows_3	43.74% <87.50%> (-0.66%)	⬇️
Windows_4	35.20% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[seraphjiang]

could we take a look configuration api in security plugin and see if we want to align the convention, and could we?

https://github.com/search?q=repo%3Aopensearch-project%2Fsecurity-dashboards-plugin+%22%2Fapi%2Fv1%2Fconfiguration%22&type=code

/api/v1/configuration/{entity}

@zengyan-amazon @Flyingliuhub @bandinib-amzn @ZilongX @BionIT

[BionIT]

Left a few comments and please let me know if any questions

[tianleh]

could we take a look configuration api in security plugin and see if we want to align the convention, and could we?

https://github.com/search?q=repo%3Aopensearch-project%2Fsecurity-dashboards-plugin+%22%2Fapi%2Fv1%2Fconfiguration%22&type=code

/api/v1/configuration/{entity}

@zengyan-amazon @Flyingliuhub @bandinib-amzn @ZilongX @BionIT

I will take a look the related links.

[tianleh]

could we take a look configuration api in security plugin and see if we want to align the convention, and could we?
https://github.com/search?q=repo%3Aopensearch-project%2Fsecurity-dashboards-plugin+%22%2Fapi%2Fv1%2Fconfiguration%22&type=code

/api/v1/configuration/{entity}

@zengyan-amazon @Flyingliuhub @bandinib-amzn @ZilongX @BionIT

I will take a look the related links.

From the public doc https://opensearch.org/docs/latest/security/access-control/api/ , the APIs have the feature name (e.g role, internalusers) in the path. This is similar to what I am proposing. E.g for csp rules, we will have APIs like this

curl -X GET 'http://localhost:5601/api/config/csp/rules'

I also mention the alternative approach in the issue.
#5796 (comment)

[seraphjiang]

From the public doc https://opensearch.org/docs/latest/security/access-control/api/ , the APIs have the feature name (e.g role, internalusers) in the path. This is similar to what I am proposing. E.g for csp rules, we will have APIs like this

curl -X GET 'http://localhost:5601/api/config/csp/rules'

I also mention the alternative approach in the issue.

Didn't see the version in endpoint of new proposal, since we already has configuration api, introduce config seem will cause confusion.

[tianleh]

From the public doc https://opensearch.org/docs/latest/security/access-control/api/ , the APIs have the feature name (e.g role, internalusers) in the path. This is similar to what I am proposing. E.g for csp rules, we will have APIs like this
curl -X GET 'http://localhost:5601/api/config/csp/rules'
I also mention the alternative approach in the issue.

Didn't see the version in endpoint of new proposal, since we already has configuration api, introduce config seem will cause confusion.

@zengyan-amazon Do you have context about adding v1 to the security OSD APIs?

@seraphjiang I don't think it is common to have version in the API. E.g this is the ISM OSD plugin APIs https://github.com/opensearch-project/index-management-dashboards-plugin/blob/76b85ea6795f2b5a9b15df660dff3a191f520544/utils/constants.ts#L6

To avoid confusion with security OSD APIs, I am proposing the new APIs to look like this

/api/appconfig/csp/rules

[tianleh]

Find a strange problem that the plugin registration is not working but my POC code is working https://github.com/tianleh/OpenSearch-Dashboards/blob/cj-dev/plugins/csp_storage_provider/server/plugin.ts#L63

This is for an external plugin to call the set function exposed from the configuration service plugin. I am still debugging it.

[tianleh]

Gain some new ideas after syncing up with @kavilla

Currently I am also experimenting an idea to add a new plugin which is responsible for implementing an OpenSearch based client and register to the configuration service. This can simplify the logic at configuration service plugin.

[tianleh]

From the public doc https://opensearch.org/docs/latest/security/access-control/api/ , the APIs have the feature name (e.g role, internalusers) in the path. This is similar to what I am proposing. E.g for csp rules, we will have APIs like this
curl -X GET 'http://localhost:5601/api/config/csp/rules'
I also mention the alternative approach in the issue.

Didn't see the version in endpoint of new proposal, since we already has configuration api, introduce config seem will cause confusion.

@zengyan-amazon Do you have context about adding v1 to the security OSD APIs?

@seraphjiang I don't think it is common to have version in the API. E.g this is the ISM OSD plugin APIs https://github.com/opensearch-project/index-management-dashboards-plugin/blob/76b85ea6795f2b5a9b15df660dff3a191f520544/utils/constants.ts#L6

To avoid confusion with security OSD APIs, I am proposing the new APIs to look like this

/api/appconfig/csp/rules

Updated both PR and RFC to reflect the new API path.

[tianleh]

Find a strange problem that the plugin registration is not working but my POC code is working https://github.com/tianleh/OpenSearch-Dashboards/blob/cj-dev/plugins/csp_storage_provider/server/plugin.ts#L63

This is for an external plugin to call the set function exposed from the configuration service plugin. I am still debugging it.

Find the root cause. For the plugin which tries to register its client, it needs to use the exact variable name which is the configuration plugin id applicationConfig when importing into its setup function.

[seraphjiang]

Gain some new ideas after syncing up with @kavilla

Thanks @kavilla and @tianleh for explore and find different way to abstract

[tianleh]

Updated the preferred approach to use a generic style API/interface #5796 (comment)

and in progress to update the PR to reflect the changes.

[kavilla]

I think it's good initial iteration. Seems pretty silo'd and I like the abstraction.

I think we can have improvements to hook into some of the code that transforms the data and make it easier to create the config index.

I think we can also update the limits.yml file but just glanced over this PR for one more time. Will look again closely later but not blocking it.