[Fleet] Use new Fleet Secrets ES APIs instead of reading/writing secrets index

[criamico]

Closes #162915

Summary

Replace direct calls to Fleet Secrets index with new API calls introduced with elastic/elasticsearch#97728

New ES secrets APIs:

POST /_fleet/secret/
{
  "value": "<secret value>"
}

// Returns the id of the created secret
{
  "id": "<secret_id>"
}

DELETE /_fleet/secret/<secret_id>

// returns 
{
  "deleted": true
}

NOTE: I tried running the secrets integration tests in #162732 but there is some ES error that I'm not sure how to address. I think that the test can be worked on separately

Testing

Testing steps are the exact same as #157176:

• Start EPR locally loading the Secrets test package from Kibana:

docker run -p 8080:8080 -v /Users/<YOUR_PATH>/kibana/x-pack/test/fleet_api_integration/apis/fixtures/test_packages:/packages/test-packages -v /Users/<YOUR_PATH>/kibana/x-pack/test/fleet_api_integration/apis/fixtures/package_registry_config.yml:/package-registry/config.yml docker.elastic.co/package-registry/package-registry:main

• Point kibana.dev.yml to local EPR:

  xpack.fleet.registryUrl: http://localhost:8080

• Enable the secrets feature flag secretsStorage
• Start kibana and navigate to integrations, install Secrets package.
• It should create and edit the package policy successfully

• The yml policy should have the redacted secrets and secrets ids:

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[criamico]

@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 5c600c9

Metrics [docs]

✅ unchanged

History

• 💔 Build #147176 failed eb30bef

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @criamico

[juliaElastic]

LGTM, I'm ok with fixing the integration tests in a follow up pr.