[Fleet] Tighten policy permissions, take II #97366

afgomez · 2021-04-16T10:45:40Z

Summary

This PR refines the work done in #94591. It narrows down the API key permissions that each agent needs based on the integrations that the policy uses.

To do so we look into the policy's PacakgePolicy and the enabled inputs. We link that with the package DataStream to determine which indices it uses and which permissions it needs. The permissions will be specified in the package's data_streams/*/manifest.yml (pending proposal).

At the moment of writing no packages specify what permissions they need, so I'm using the permissions that are necessary right now as defaults.

Testing

To test this PR:

Go to Fleet > Policies
Click on any policy
Click on Actions > View policy

The policy should have an output_permissions block, with several permissions per package instead of the generic logs-*, metrics-*, etc.

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

Prepare code to provide tighth permissions as well.

We can read the package information from two places - The EPM registry (method `getRegistryPackage()`) - The local ES cache (method `getEsPackage()`) The package contains a property called `data_streams`. The contents of this property varied depending on from where the package was being loaded. The code in `getEsPackage` was cherry-picking what properties to load to do validation. We have changed the code so we cherry-pick only the properties that need some sort of validation, and pass the others in bulk.

elasticmachine · 2021-05-25T13:22:28Z

Pinging @elastic/fleet (Feature:Fleet)

ruflin · 2021-05-25T13:50:32Z

My assumption is that as long as a package does not specify any more fine grained permissions the permissions granted would be to append data for the data streams specified in the package as a default. Is this part of this PR?

afgomez · 2021-05-25T15:37:23Z

My assumption is that as long as a package does not specify any more fine grained permissions the permissions granted would be to append data for the data streams specified in the package as a default. Is this part of this PR?

@ruflin yes, and only for the enabled data streams.

For example, if a user adds the apache integration with only logs (no metrics), it will only give permissions to that agent to logs-apache-{namespace}.

jen-huang · 2021-05-26T15:22:02Z

Hi @afgomez, in addition to the edge cases we discussed (Custom Logs, Endpoint, APM, etc), I remembered another set of permissions we need to add. If agent monitoring is enabled, we need to add permissions for those indices: elastic/beats#25350 (comment).

monitoring_enabled on agent policy can contain logs and/or metrics or both, so we need to add permissions for whatever is enabled, i.e. logs-elastic_agent.*-{AGENT POLICY NAMESPACE} and metrics-elastic_agent.*-{AGENT POLICY NAMESPACE} (I think those are the right patterns, but you might want to double check with agent team).

afgomez · 2021-06-03T08:04:46Z

One more thought: as it currently stands, users won't get these new agent policies with permissions until a "real" change is triggered on their policies (from any user update on the policy, or a global Settings change)

I'm assuming the agent releases are tied to the rest of the stack. How does it work right now when the user upgrades the stack?

A user upgrades kibana to 7.14
Then it upgrades the agents to 7.14

What happens in the second step? Will the agent fetch its policy then?

afgomez · 2021-06-03T09:52:23Z

@elasticmachine merge upstream

x-pack/plugins/fleet/server/services/agent_policy.ts

x-pack/plugins/fleet/server/services/package_policies_to_agent_permissions.ts

kibanamachine · 2021-06-08T13:04:48Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	981	985	+4

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	425.1KB	425.2KB	+49.0B

Unknown metric groups

API count

id	before	after	diff
`fleet`	1071	1075	+4

History

💚 Build #129442 succeeded f40569c
💚 Build #128861 succeeded 1dc4ad5
💔 Build #128836 failed 55383a9
💔 Build #128635 failed c7bff76
💔 Build #128614 failed 1d07565

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @afgomez

jen-huang

Thank you for making all the changes! I tested locally, Kibana only didn't test with agent usage yet, but the agent YAML LGTM 🚀

jen-huang · 2021-06-08T23:01:50Z

x-pack/plugins/fleet/server/services/agent_policy.ts

+      cluster: DEFAULT_PERMISSIONS.cluster,
+    };
+
+    // TODO fetch this from the elastic agent package


Could you file an issue for this and link elastic/integrations#953 as a blocker?

ruflin · 2021-06-09T06:08:56Z

One thing that would be nice to do on this PR would be to run the e2e-tests once before merging, it should show if most permissions we need are still in place. If I remember correctly this is possible. @mdelapenya should be able to help here.

mdelapenya · 2021-06-09T06:42:25Z

/run-fleet-e2e-tests

mdelapenya · 2021-06-09T06:46:20Z

One thing that would be nice to do on this PR would be to run the e2e-tests once before merging, it should show if most permissions we need are still in place. If I remember correctly this is possible. @mdelapenya should be able to help here.

I started the tests with the above command. The result is:

Kibana will be built for this PR: https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-kibana-fleet/detail/e2e-testing-kibana-fleet/28/pipeline
Once built, tests will use docker.elastic.co/observability-ci/kibana:pr97366 as image (will update this comment with the link to the job, as we did not decide how to contribute back the CI notifications)

mdelapenya · 2021-06-09T12:50:16Z

Once built, tests will use docker.elastic.co/observability-ci/kibana:pr97366 as image (will update this comment with the link to the job, as we did not decide how to contribute back the CI notifications)

The job finished with errors: https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/master/965/tests

ARM errors: do not pay attention to them, as we discovered the kibana image for PRs is not built on ARM (work in progress)
There are 3 potential errors:
- Deploying an Endpoint makes policies to appear in the Security App Change of links to styles and js #2 – Agent Endpoint Integration
- Changing an Agent policy is reflected in the Security App links to styles and js #1 – Agent Endpoint Integration
- Changing an Agent policy is reflected in the Security App Change of links to styles and js #2 – Agent Endpoint Integration

All three fail because of timeouts: not reaching a desired state during a time window defined by max timeout (5-10 minutes!). The first one passes for Centos but not for Debian.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-06-09T14:13:43Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Alejandro Fernández Gómez <alejandro.fernandez@elastic.co>

* master: clarify which parts of TM are experimental (elastic#101757) Add sh scripts with _bulk_action route usage examples (elastic#101736) [Uptime] Only register route in side nav if uptime show capability is true (elastic#101709) Use KIBANA_DOCS in doc link service (elastic#101667) [Alerting][Event log] Persisting duration information for active alerts in event log (elastic#101387) Address design issues in Discover/Graph (elastic#101584) Optimize performance for document table (elastic#101715) Change file data visualizer links to point to new location in home application (elastic#101393) [Fleet] Tighten policy permissions, take II (elastic#97366) [ML] Add debounce to the severity control update (elastic#101581) [Fleet] Fix routing issues with `getPath` and `history.push` (elastic#101658) [APM] Add link-to/transaction route (elastic#101731) [Index Patterns] Runtime fields CRUD REST API (elastic#101164) [ILM] Refactor types and fix missing aria labels (elastic#101518) [Lens] New summary row feature for datatable (elastic#101075) Blocks save event filter with a white space name (elastic#101599) Improve security server types (elastic#101661) [APM] Replace side nav with tabs on Settings page (elastic#101460) [APM] Only register items in side nav if user has permissions to see app (elastic#101707) [Security solution][Endpoint] Add back button when to the event filters list (elastic#101280)

afgomez added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:Fleet Fleet team's agent central management project Team:Fleet Team label for Observability Data Collection Fleet team v7.13.0 labels Apr 16, 2021

afgomez force-pushed the 64634-policy-permissions-per-package-policy branch from 3e552e0 to c17f064 Compare April 16, 2021 10:49

afgomez mentioned this pull request Apr 16, 2021

[Fleet] Tighten policy permissions #96063

Closed

2 tasks

ruflin added v7.14.0 and removed v7.13.0 labels Apr 18, 2021

afgomez force-pushed the 64634-policy-permissions-per-package-policy branch 3 times, most recently from 368748d to 8b1426f Compare May 10, 2021 13:59

Alejandro Fernández Gómez added 8 commits May 25, 2021 15:22

Add config flag to enable the new permission block

a0cbb98

Refactor default permission handling

9272808

Prepare code to provide tighth permissions as well.

Add permissiosn key to package data_streams

b6bb88a

Get permissions from active package policies

61ce8f7

Send default cluster permissions as a separate integration

0ba163f

Clarify type naming

3c95ad1

Add specs

dcc4c47

afgomez force-pushed the 64634-policy-permissions-per-package-policy branch from 8b1426f to dcc4c47 Compare May 25, 2021 13:22

afgomez marked this pull request as ready for review May 25, 2021 13:22

afgomez requested a review from a team as a code owner May 25, 2021 13:22

Add missing property to the mock

948aa5f

jen-huang assigned afgomez May 25, 2021

jen-huang self-requested a review May 25, 2021 17:20

afgomez requested a review from jen-huang June 3, 2021 08:01

kibanamachine and others added 3 commits June 3, 2021 05:52

Merge branch 'master' into 64634-policy-permissions-per-package-policy

1dc4ad5

Handle permissions for agent monitoring

3d8d2e1

Merge branch 'master' into 64634-policy-permissions-per-package-policy

f40569c

jen-huang reviewed Jun 7, 2021

View reviewed changes

x-pack/plugins/fleet/server/services/agent_policy.ts Outdated Show resolved Hide resolved

jen-huang reviewed Jun 7, 2021

View reviewed changes

x-pack/plugins/fleet/server/services/package_policies_to_agent_permissions.ts Outdated Show resolved Hide resolved

Alejandro Fernández Gómez added 3 commits June 8, 2021 12:55

Use the right patterns for monitoring

853023d

Refine typing

340999b

Merge branch 'master' into 64634-policy-permissions-per-package-policy

7a237ef

jen-huang self-requested a review June 8, 2021 16:58

jen-huang approved these changes Jun 8, 2021

View reviewed changes

jen-huang added release_note:enhancement auto-backport Deprecated - use backport:version if exact versions are needed and removed release_note:skip Skip the PR/issue when compiling release notes labels Jun 8, 2021

afgomez merged commit cd8cf46 into elastic:master Jun 9, 2021

afgomez deleted the 64634-policy-permissions-per-package-policy branch June 9, 2021 14:11

kibanamachine added a commit to kibanamachine/kibana that referenced this pull request Jun 9, 2021

[Fleet] Tighten policy permissions, take II (elastic#97366)

e82c11e

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine mentioned this pull request Jun 9, 2021

[7.x] [Fleet] Tighten policy permissions, take II (#97366) #101766

Merged

axw mentioned this pull request Jun 18, 2021

[tail sampling] make sure apm-server has enough privileges for tail-based sampling elastic/apm-server#5490

Closed

axw mentioned this pull request Aug 1, 2021

[Change Proposal] Add support for specifying Elasticsearch privileges elastic/package-spec#203

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Tighten policy permissions, take II #97366

[Fleet] Tighten policy permissions, take II #97366

afgomez commented Apr 16, 2021 •

edited

Loading

elasticmachine commented May 25, 2021

ruflin commented May 25, 2021

afgomez commented May 25, 2021 •

edited

Loading

jen-huang commented May 26, 2021

afgomez commented Jun 3, 2021

afgomez commented Jun 3, 2021

kibanamachine commented Jun 8, 2021

API count

jen-huang left a comment

jen-huang Jun 8, 2021

ruflin commented Jun 9, 2021

mdelapenya commented Jun 9, 2021

mdelapenya commented Jun 9, 2021 •

edited

Loading

mdelapenya commented Jun 9, 2021

kibanamachine commented Jun 9, 2021

[Fleet] Tighten policy permissions, take II #97366

[Fleet] Tighten policy permissions, take II #97366

Conversation

afgomez commented Apr 16, 2021 • edited Loading

Summary

Testing

Checklist

elasticmachine commented May 25, 2021

ruflin commented May 25, 2021

afgomez commented May 25, 2021 • edited Loading

jen-huang commented May 26, 2021

afgomez commented Jun 3, 2021

afgomez commented Jun 3, 2021

kibanamachine commented Jun 8, 2021

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Page load bundle

API count

History

jen-huang left a comment

Choose a reason for hiding this comment

jen-huang Jun 8, 2021

Choose a reason for hiding this comment

ruflin commented Jun 9, 2021

mdelapenya commented Jun 9, 2021

mdelapenya commented Jun 9, 2021 • edited Loading

mdelapenya commented Jun 9, 2021

kibanamachine commented Jun 9, 2021

💚 Backport successful

afgomez commented Apr 16, 2021 •

edited

Loading

afgomez commented May 25, 2021 •

edited

Loading

mdelapenya commented Jun 9, 2021 •

edited

Loading