Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core.logging] Add ops logs to the KP logging system #88070

Merged
merged 37 commits into from
Jan 27, 2021

Conversation

TinaHeiligers
Copy link
Contributor

@TinaHeiligers TinaHeiligers commented Jan 12, 2021

Resolves #86162

Summary

Logs ops metrics data from the Kibana platform using the core logger for feature parity with the legacy logging server.
The legacy logging server relies on the @hapi/good ops event for automatically logging ops metrics.

The ops metrics are logged from within the metrics_service, with a message format that mimics the legacy logging format (e.g: memory: 221.7MB uptime: 0:10:40 load: [8.06,7.07,13.33] delay: 0.259).
In addition, we're added ECS-compatible structured LogMeta, and used custom fields for data that isn't (yet) a standard ECS field.

The ops metrics are logged out when the ops metrics refresh (i.e. at the configured ops.interval).

Changes

  • Adds logging to metrics server.
  • Adds a utility for extracting the ops data.
  • Adds config deprecations for logging.events.ops & updates docs.
  • Disables legacy ops logs when logging.verbose: true, as this results in duplicate data with the new metrics logs.

Plugin API Changes

We are deprecating the legacy response logs which were enabled when logging.verbose: true or when using logging.events.ops. It will be removed completely in 8.0, and has been replaced with new ops logs that is provided under the metrics.ops context at the debug level:

Before

logging:
  events:
    ops: "*"

After

logging:
  loggers:
    - context: metrics.ops
      appenders: [console]
      level: debug

For more information, check out the section on logging config migration in the logging README.

How to test this:

  1. Add the following logging configuration to your kibana.yml file:
#kibana.yml or kibana.dev.yml
logging:
  events:
    log: ['debug']
    json: false
    verbose: true

  appenders:
    console:
      kind: console
      layout:
        kind: pattern
        highlight: true

  root:
    appenders: [default]
    level: warn

  loggers:
    - context: metrics.metrics
      appenders: [console]
      level: debug
  1. Start es and kibana.
  2. Observe that the ops metrics are logged out (std out): e.g.
[2021-01-20T22:30:06.974Z][DEBUG][metrics.ops]{"ecs":{"version":"1.7.0"},"kind":"metric","category":["process","host"],"process":{"uptime":640,"memory":{"heap":{"usedInBytes":232472872}},"eventLoopDelay":0.25925004482269287},"host":{"os":{"load":{"1m":8.0625,"5m":7.07470703125,"15m":13.32568359375}}}} memory: 221.7MB uptime: 0:10:40 load: [8.06,7.07,13.33] delay: 0.259

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@TinaHeiligers TinaHeiligers added Feature:Logging release_note:enhancement Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.12.0 v8.0.0 labels Jan 12, 2021
@TinaHeiligers
Copy link
Contributor Author

@lukeelmers Could you please take a look over the implementation and the tests added to log the ops metrics from the KP?
I'll work on the documentation in the mean time and change the PR to Ready for review if you're happy with it.

src/core/server/metrics/metrics_service.test.ts Outdated Show resolved Hide resolved
src/core/server/metrics/metrics_service.ts Outdated Show resolved Hide resolved
src/core/server/metrics/metrics_service.ts Outdated Show resolved Hide resolved
@TinaHeiligers
Copy link
Contributor Author

@elasticmachine merge upstream

@TinaHeiligers
Copy link
Contributor Author

@elasticmachine merge upstream

@TinaHeiligers TinaHeiligers changed the title POC for KP ops metrics logging [core.logging] Add ops logs to the KP logging system Jan 20, 2021
@TinaHeiligers TinaHeiligers added release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. and removed release_note:enhancement labels Jan 20, 2021
@TinaHeiligers
Copy link
Contributor Author

@elasticmachine merge upstream

Copy link
Contributor

@mshustov mshustov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a couple of nits

src/core/server/logging/README.md Show resolved Hide resolved
src/core/server/logging/ecs.ts Outdated Show resolved Hide resolved
@TinaHeiligers
Copy link
Contributor Author

@elasticmachine merge upstream

@TinaHeiligers
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@TinaHeiligers TinaHeiligers merged commit c6cfdee into elastic:master Jan 27, 2021
@TinaHeiligers TinaHeiligers deleted the logging/ops-metrics branch January 27, 2021 03:23
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jan 27, 2021
…y-tests

* 'master' of github.com:elastic/kibana: (276 commits)
  [Telemetry] Settings Collector: redact sensitive reported values (elastic#88675)
  [CI] Combines Jest test jobs (elastic#85850)
  [Upgrade Assistant] Migrate server to new es-js client (elastic#89207)
  Migrate maps_legacy, maps_oss, region_map, and tile_map plugions to TS projects (elastic#89351)
  [Vega Docs] Add experimental flag on the vega maps title (elastic#89402)
  Increase the time needed to locate the save viz toast (elastic#89301)
  [Enterprise Search] Add links to doc links service (elastic#89260)
  Fixed regex bug in Safari (elastic#89399)
  [Lens] Fix indexpattern checks for missing references (elastic#88840)
  [Lens] Clean up usage collector (elastic#89109)
  update apm index pattern (elastic#89395)
  [APM] Upgrade ES client (elastic#86594)
  Enable v2 so migrations, disable in FTR tests (elastic#89297)
  [Search Sessions] Make search session indicator UI opt-in, refactor per-app capabilities (elastic#88699)
  Cleanup OSS code from visualizations wizard (elastic#89092)
  [APM] Optimize API test order (elastic#88654)
  Rename conversion function, extract to module scope and add tests. (elastic#89018)
  [core.logging] Add ops logs to the KP logging system (elastic#88070)
  chore(NA): improve ts build refs performance on kbn bootstrap (elastic#89333)
  skip flaky suite (elastic#89379)
  ...

# Conflicts:
#	x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/timeline/timeline.tsx
#	x-pack/test/accessibility/config.ts
gmmorris added a commit to gmmorris/kibana that referenced this pull request Jan 27, 2021
…ana into task-manager/shift-on-trend

* 'task-manager/shift-on-trend' of github.com:gmmorris/kibana: (74 commits)
  [Metrics UI] Fix Host Overview boxes in Host Detail page (elastic#89299)
  [Telemetry] Settings Collector: redact sensitive reported values (elastic#88675)
  [CI] Combines Jest test jobs (elastic#85850)
  [Upgrade Assistant] Migrate server to new es-js client (elastic#89207)
  Migrate maps_legacy, maps_oss, region_map, and tile_map plugions to TS projects (elastic#89351)
  [Vega Docs] Add experimental flag on the vega maps title (elastic#89402)
  Increase the time needed to locate the save viz toast (elastic#89301)
  [Enterprise Search] Add links to doc links service (elastic#89260)
  Fixed regex bug in Safari (elastic#89399)
  [Lens] Fix indexpattern checks for missing references (elastic#88840)
  [Lens] Clean up usage collector (elastic#89109)
  update apm index pattern (elastic#89395)
  [APM] Upgrade ES client (elastic#86594)
  Enable v2 so migrations, disable in FTR tests (elastic#89297)
  [Search Sessions] Make search session indicator UI opt-in, refactor per-app capabilities (elastic#88699)
  Cleanup OSS code from visualizations wizard (elastic#89092)
  [APM] Optimize API test order (elastic#88654)
  Rename conversion function, extract to module scope and add tests. (elastic#89018)
  [core.logging] Add ops logs to the KP logging system (elastic#88070)
  chore(NA): improve ts build refs performance on kbn bootstrap (elastic#89333)
  ...
TinaHeiligers added a commit that referenced this pull request Jan 27, 2021
…89419)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Logging release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.12.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[logging/metrics] Provide ability to log ops data from core metrics service
6 participants