Remove dependency on Fleet Default policy

[juliaElastic]

Proposal

Fleet is making a change to remove Default policies from setup, in order to make policy creation with integrations explicit: elastic/kibana#108456
It seems that ECK relies on this Default policy currently, we request to make a change to create a dedicated policy first, and use that for ECK setup instead of expecting a default to be present.

Here is an example on creating a policy with API:

POST kibana_host/api/fleet/agent_policies?sys_monitoring=true
kbn-xsrf: kibana

{"name":"Agent policy 1","namespace":"default","monitoring_enabled":["logs","metrics"]}

Alternatively the policy can be added to kibana.yml config as preconfiguration:

xpack.fleet.packages:
  - name: system
    version: latest
  - name: elastic_agent
    version: latest
xpack.fleet.agentPolicies:
  - name: Agent policy 1
    description: Agent policy 1
    is_managed: false
    namespace: default
    monitoring_enabled:
      - logs
      - metrics
    package_policies:
      - name: system-1
        id: default-system
        package:
          name: system

[joshdover]

Alternatively the policy can be added to kibana.yml config as preconfiguration:

This isn't the recommended approach but I think this would likely be the most viable option in the short-term since it's simply adding a value to kibana.yml. If you'd like, this can always be present and you'd get the same behavior as 8.0 and prior releases.

Ideally, we use an API call to create this only when a Fleet Server or Elastic Agent is added to avoid creating assets on the Kibana instance that aren't used by the user, which is the overall goal of this change to remove these default assets.

Since ECK supports both Fleet Server resources and Elastic Agent running as a daemon set for metric collection, I think there are two policies that would need to be created:

xpack.fleet.packages:
  - name: system
    version: latest
  - name: elastic_agent
    version: latest
  - name: fleet_server
    version: latest
xpack.fleet.agentPolicies:
  - name: ECK Fleet Server policy
    id: eck-fleet-server
    description: Fleet Server policy
    is_managed: false
    namespace: default
    monitoring_enabled:
      - logs
      - metrics
    package_policies:
      - name: fleet_server-1
        id: default-fleet-server
        package:
          name: fleet_server
  - name: ECK Agent
    id: eck-agent
    description: Agent policy 1
    is_managed: false
    namespace: default
    monitoring_enabled:
      - logs
      - metrics
    package_policies:
      - name: system-1
        id: default-system
        package:
          name: system

[david-kow]

I think it should suffice to modify our examples and E2E tests when we move to the version that will introduce this change. We could treat this as any other change between versions of the products we orchestrate.

[joshdover]

I think it should suffice to modify our examples and E2E tests when we move to the version that will introduce this change. We could treat this as any other change between versions of the products we orchestrate.

That's good to hear. Does that just mean updating the examples in the docs like this? https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-elastic-agent-fleet-quickstart.html

If so, I think we could move ahead with this change now. The Fleet UI team would be happy to contribute updates to these docs & examples, if needed.

[david-kow]

Yes, that's correct. Note that we also have https://github.com/elastic/cloud-on-k8s/tree/main/config/recipes/elastic-agent which is linked from the docs. The difference here is that those recipes are E2E tested - we would need to have those tests passing before merging (this can be tested in a PR by commenting run full pr build).

[joshdover]

@juliaElastic do you have capacity to take on these updates?

[juliaElastic]

@joshdover yes, so to update the doc above and the recipes, right?

[naemono]

@juliaElastic yes, and comment on the PR run full pr build, and ensure the check that is created in github succeeds.