Skip to content
This repository has been archived by the owner on Sep 17, 2024. It is now read-only.

Remove dependency on Fleet Default policy #2039

Closed
juliaElastic opened this issue Jan 20, 2022 · 14 comments · Fixed by #2111
Closed

Remove dependency on Fleet Default policy #2039

juliaElastic opened this issue Jan 20, 2022 · 14 comments · Fixed by #2111
Labels
enhancement New feature or request

Comments

@juliaElastic
Copy link
Contributor

juliaElastic commented Jan 20, 2022

Proposal

Fleet is making a change to remove Default policies from setup, in order to make policy creation with integrations explicit: elastic/kibana#108456

Some of the e2e tests seem to rely on default policies:

func (c *Client) GetDefaultPolicy(ctx context.Context, fleetServer bool) (Policy, error) {

The request is to change this logic to create required policies first through the API or preconfiguration.

EDIT: GetDefaultPolicy function does not seem to be used, and there was an issue some time back to remove the reference on default policy: #281
So it might not be needed to change anything, the request is to confirm that nothing breaks.

Here is an example on creating a policy with API:

POST kibana_host/api/fleet/agent_policies?sys_monitoring=true
kbn-xsrf: kibana

{"name":"Agent policy 1","namespace":"default","monitoring_enabled":["logs","metrics"]}

Alternatively the policy can be added to kibana.yml config as preconfiguration:

xpack.fleet.packages:
  - name: system
    version: latest
  - name: elastic_agent
    version: latest
xpack.fleet.agentPolicies:
  - name: Agent policy 1
    description: Agent policy 1
    is_managed: false
    namespace: default
    monitoring_enabled:
      - logs
      - metrics
    package_policies:
      - name: system-1
        id: default-system
        package:
          name: system

Example preconfig for Fleet Server policy:

xpack.fleet.packages:
    - name: fleet_server
      version: latest
xpack.fleet.agentPolicies:
    - name: Fleet Server policy
      id: fleet-server-policy
      description: Fleet server policy
      namespace: default
      package_policies:
          - name: Fleet Server
            package:
                name: fleet_server

And use it in docker command or elastic-agent install command to enroll Fleet Server:
-e FLEET_SERVER_POLICY_ID=fleet-server-policy
--fleet-server-policy=fleet-server-policy

@juliaElastic
Copy link
Contributor Author

juliaElastic commented Jan 31, 2022

@mdelapenya @adam-stokes hello, could someone confirm that e2e tests are not relying on the default agent policies in Fleet?
I've tried to run the Fleet suite with my kibana pr, but it's failing with a dependency error:
https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-kibana-fleet/54/console

11:50:51  ERROR [bootstrap] failed:
11:50:51  ERROR The bazel command that was running failed to complete.
11:50:51  error Command failed with exit code 1.
11:50:51  info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
11:50:51  Error: Cannot find module '@kbn/optimizer'
11:50:51  Require stack:
11:50:51  - /var/lib/jenkins/workspace/e2e-tests/e2e-testing-kibana-fleet/src/github.com/elastic/kibana/src/setup_node_env/index.js
11:50:51  - /var/lib/jenkins/workspace/e2e-tests/e2e-testing-kibana-fleet/src/github.com/elastic/kibana/scripts/build.js
11:50:51      at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
11:50:51      at Module.Hook._require.Module.require (/var/lib/jenkins/workspace/e2e-tests/e2e-testing-kibana-fleet/src/github.com/elastic/kibana/node_modules/require-in-the-middle/index.js:61:29)
11:50:51      at require (node:internal/modules/cjs/helpers:102:18)
11:50:51      at Object.<anonymous> (/var/lib/jenkins/workspace/e2e-tests/e2e-testing-kibana-fleet/src/github.com/elastic/kibana/src/setup_node_env/index.js:11:1)
11:50:51      at Module._compile (node:internal/modules/cjs/loader:1101:14)
11:50:51      at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
11:50:51      at Module.load (node:internal/modules/cjs/loader:981:32)
11:50:51      at Function.Module._load (node:internal/modules/cjs/loader:822:12)
11:50:51      at Module.require (node:internal/modules/cjs/loader:1005:19)
11:50:51      at require (node:internal/modules/cjs/helpers:102:18)
11:50:53  [Pipeline] sleep
11:50:53  Sleeping for 10 sec
11:51:03  [Pipeline] }
11:51:03  ERROR: script returned exit code 1

@mdelapenya
Copy link
Contributor

I'm looking at the code and will post here my findings, but my initial thoughts are with you: we are creating a policy before each test scenario so that it's the one used by the agent for any operation.

I'd say it's OK to merge that PR, and if it breaks the e2e, let's cooperate in the fix 😃

@mdelapenya
Copy link
Contributor

@mdelapenya @adam-stokes hello, could someone confirm that e2e tests are not relying on the default agent policies in Fleet? I've tried to run the Fleet suite with my kibana pr, but it's failing with a dependency error: https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-kibana-fleet/54/console

11:50:51  ERROR [bootstrap] failed:
11:50:51  ERROR The bazel command that was running failed to complete.
11:50:51  error Command failed with exit code 1.
11:50:51  info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
11:50:51  Error: Cannot find module '@kbn/optimizer'
11:50:51  Require stack:
11:50:51  - /var/lib/jenkins/workspace/e2e-tests/e2e-testing-kibana-fleet/src/github.com/elastic/kibana/src/setup_node_env/index.js
11:50:51  - /var/lib/jenkins/workspace/e2e-tests/e2e-testing-kibana-fleet/src/github.com/elastic/kibana/scripts/build.js
11:50:51      at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
11:50:51      at Module.Hook._require.Module.require (/var/lib/jenkins/workspace/e2e-tests/e2e-testing-kibana-fleet/src/github.com/elastic/kibana/node_modules/require-in-the-middle/index.js:61:29)
11:50:51      at require (node:internal/modules/cjs/helpers:102:18)
11:50:51      at Object.<anonymous> (/var/lib/jenkins/workspace/e2e-tests/e2e-testing-kibana-fleet/src/github.com/elastic/kibana/src/setup_node_env/index.js:11:1)
11:50:51      at Module._compile (node:internal/modules/cjs/loader:1101:14)
11:50:51      at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
11:50:51      at Module.load (node:internal/modules/cjs/loader:981:32)
11:50:51      at Function.Module._load (node:internal/modules/cjs/loader:822:12)
11:50:51      at Module.require (node:internal/modules/cjs/loader:1005:19)
11:50:51      at require (node:internal/modules/cjs/helpers:102:18)
11:50:53  [Pipeline] sleep
11:50:53  Sleeping for 10 sec
11:51:03  [Pipeline] }
11:51:03  ERROR: script returned exit code 1

This seems an error in the kibana build, which I'm not familiar with. We created #2086 to address the changes in the requirements to run kibana in the e2e tests, which does not need ARM. Is this your use case?

@juliaElastic
Copy link
Contributor Author

@mdelapenya no, I don't need ARM specifically

@mdelapenya
Copy link
Contributor

mdelapenya commented Feb 8, 2022

Thanks @juliaElastic for your explanation here. Now that #2064 is merged, the GetDefaultPolicy method is used to retrieve the default fleet-server policy before bootstrapping it.

I'll create a PR to pre-configure the policy with:

xpack.fleet.packages:
    - name: fleet_server-1
      version: latest
xpack.fleet.agentPolicies:
    - name: Fleet Server policy
      id: fleet-server-policy
      description: Fleet server policy
      namespace: default
      package_policies:
          - name: Fleet Server
            package:
                name: fleet_server-1

Thanks!

@mdelapenya
Copy link
Contributor

After adding that, I'm receiving this error while setting up Fleet:

WARN[2022-02-08T09:47:14+01:00] Fleet not ready                               body="{\"error\":\"Internal Server Error\",\"message\":\"[Fleet Server policy] could not be added. [fleet_server-1] is not installed, add [fleet_server-1] to [xpack.fleet.packages] or remove it from [Fleet Server].\",\"statusCode\":500}" statusCode=500

@juliaElastic
Copy link
Contributor Author

After adding that, I'm receiving this error while setting up Fleet:

WARN[2022-02-08T09:47:14+01:00] Fleet not ready                               body="{\"error\":\"Internal Server Error\",\"message\":\"[Fleet Server policy] could not be added. [fleet_server-1] is not installed, add [fleet_server-1] to [xpack.fleet.packages] or remove it from [Fleet Server].\",\"statusCode\":500}" statusCode=500

@mdelapenya oh sorry, the example was not correct, packages.name has to be the same as agentPolicies.package_policies.name, like this:

xpack.fleet.packages:
    - name: fleet_server-1
      version: latest

@mdelapenya
Copy link
Contributor

mdelapenya commented Feb 8, 2022

After updating to that value, the error happens on Kibana side:

[2022-02-08T09:20:05.396+00:00][WARN ][plugins.fleet] Failed installing package [fleet_server-1] due to error: [PackageNotFoundError: fleet_server-1 not found]
[2022-02-08T09:20:09.521+00:00][ERROR][plugins.fleet] Error: [Fleet Server policy] could not be added. [fleet_server-1] could not be installed due to error: [PackageNotFoundError: fleet_server-1 not found]
    at /usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:293:19
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at ensurePreconfiguredPackagesAndPolicies (/usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:275:40)
    at createSetupSideEffects (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup.js:88:7)
    at awaitIfPending (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup_utils.js:38:20)
    at /usr/share/kibana/x-pack/plugins/fleet/server/plugin.js:232:9
[2022-02-08T09:20:09.523+00:00][WARN ][plugins.fleet] Fleet setup failed
[2022-02-08T09:20:09.523+00:00][WARN ][plugins.fleet] Error: [Fleet Server policy] could not be added. [fleet_server-1] could not be installed due to error: [PackageNotFoundError: fleet_server-1 not found]
    at /usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:293:19
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at ensurePreconfiguredPackagesAndPolicies (/usr/share/kibana/x-pack/plugins/fleet/server/services/preconfiguration.js:275:40)
    at createSetupSideEffects (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup.js:88:7)
    at awaitIfPending (/usr/share/kibana/x-pack/plugins/fleet/server/services/setup_utils.js:38:20)
    at /usr/share/kibana/x-pack/plugins/fleet/server/plugin.js:232:9
[2022-02-08T09:20:09.545+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
[2022-02-08T09:20:12.385+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2022-02-08T09:20:12.890+00:00][WARN ][plugins.fleet] Failed installing package [fleet_server-1] due to error: [PackageNotFoundError: fleet_server-1 not found]
[2022-02-08T09:20:12.951+00:00][INFO ][plugins.fleet] Encountered non fatal errors during Fleet setup
[2022-02-08T09:20:12.951+00:00][INFO ][plugins.fleet] {"name":"PackageNotFoundError","message":"fleet_server-1 not found"}
[2022-02-08T09:20:12.952+00:00][INFO ][plugins.fleet] Fleet setup completed
[2022-02-08T09:20:15.220+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2022-02-08T09:20:15.715+00:00][WARN ][plugins.fleet] Failed installing package [fleet_server-1] due to error: [PackageNotFoundError: fleet_server-1 not found]
[2022-02-08T09:20:15.766+00:00][INFO ][plugins.fleet] Encountered non fatal errors during Fleet setup
[2022-02-08T09:20:15.766+00:00][INFO ][plugins.fleet] {"name":"PackageNotFoundError","message":"fleet_server-1 not found"}
[2022-02-08T09:20:15.766+00:00][INFO ][plugins.fleet] Fleet setup completed
[2022-02-08T09:20:15.778+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2022-02-08T09:20:16.278+00:00][WARN ][plugins.fleet] Failed installing package [fleet_server-1] due to error: [PackageNotFoundError: fleet_server-1 not found]
[2022-02-08T09:20:16.329+00:00][INFO ][plugins.fleet] Encountered non fatal errors during Fleet setup
[2022-02-08T09:20:16.329+00:00][INFO ][plugins.fleet] {"name":"PackageNotFoundError","message":"fleet_server-1 not found"}
[2022-02-08T09:20:16.329+00:00][INFO ][plugins.fleet] Fleet setup completed

I'm currently testing with package name = fleet_server

@juliaElastic
Copy link
Contributor Author

@mdelapenya please try with fleet_server in both places, without -1.

@mdelapenya
Copy link
Contributor

Yes, now the initialisation of Kibana works with:

xpack.fleet.packages:
  - name: fleet_server
    version: latest
xpack.fleet.agentPolicies:
  - name: Test preconfigured policy
    id: "test-preconfigured"
    data_output_id: output-123
    monitoring_output_id: output-123
    monitoring_enabled: ["logs", "metrics"]
    package_policies:
      - package:
          name: system
        name: sytem-1
  - name: Fleet Server policy
    id: fleet-server-policy
    description: Fleet server policy
    namespace: default
    package_policies:
      - name: Fleet Server
        package:
          name: fleet_server

But the Fleet Server setup fails:

TRAC[2022-02-08T10:24:30+01:00] Kibana API Query                              body=null method=GET url="http://localhost:5601/api/fleet/agents/setup"
WARN[2022-02-08T10:24:30+01:00] Fleet is not ready                            body="{\"isReady\":false,\"missing_requirements\":[\"fleet_server\"]}" error="<nil>" statusCode=200

@juliaElastic
Copy link
Contributor Author

@mdelapenya it doesn't mean the setup failed, only that there is no fleet server connected. Can you try enrolling a fleet server?

@mdelapenya
Copy link
Contributor

The code automatically adds the fleet-server passing the Policy ID on server's startup.

These are the server logs:

erforming setup of Fleet in Kibana

Policy selected for enrollment:  fleet-server-policy
{"log.level":"info","@timestamp":"2022-02-08T09:23:39.213Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":572},"message":"Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:39.365Z","log.origin":{"file.name":"application/application.go","file.line":68},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:39.366Z","log.origin":{"file.name":"application/application.go","file.line":88},"message":"Agent is in Fleet Server bootstrap mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:39.554Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":62},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:39.554Z","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":130},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:39.555Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":64},"message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:39.555Z","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":140},"message":"Agent is stopped","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:40.216Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":744},"message":"Waiting for Elastic Agent to start Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:41.601Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is HaKYwnPA","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:41.602Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 1 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:42.047Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-02-08T09:23:42Z - message: Application: fleet-server--8.1.0-SNAPSHOT[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:42.048Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:42.218Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":777},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:43.079Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-02-08T09:23:43Z - message: Application: fleet-server--8.1.0-SNAPSHOT[]: State changed to STARTING: Waiting on fleet-server input to be added to policy: fleet-server-policy - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:23:46.223Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":777},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:25:39.080Z","log.origin":{"file.name":"cmd/run.go","file.line":185},"message":"Shutting down Elastic Agent and sending last events...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:25:39.080Z","log.origin":{"file.name":"operation/operator.go","file.line":216},"message":"waiting for installer of pipeline 'default' to finish","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:25:39.080Z","log.origin":{"file.name":"process/app.go","file.line":176},"message":"Signaling application to stop because of shutdown: fleet-server--8.1.0-SNAPSHOT","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:25:40.583Z","log.origin":{"file.name":"cmd/run.go","file.line":193},"message":"Shutting down completed.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:25:40.583Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-02-08T09:25:40Z - message: Application: fleet-server--8.1.0-SNAPSHOT[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-02-08T09:25:40.583Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":66},"message":"Stats endpoint (/usr/share/elastic-agent/state/data/tmp/elastic-agent.sock) finished: accept unix /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock: use of closed network connection","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.1/fleet-troubleshooting.html
Error: enrollment failed: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.1/fleet-troubleshooting.html

It seems the policy does not have the input:

{"log.level":"info","@timestamp":"2022-02-08T09:23:46.223Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":777},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}

@juliaElastic
Copy link
Contributor Author

@mdelapenya which version of kibana are you using? we fixed a bug like this recently: elastic/kibana#124363

@mdelapenya
Copy link
Contributor

mdelapenya commented Feb 8, 2022

In our local tests, we are using 8.1.0-aa69d697-SNAPSHOT, although I can try with a more recent one, like 8.1.0-98e72485-SNAPSHOT

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants