[logging] Format new platform json logging to ECS #71138

jbudz · 2020-07-08T18:28:48Z

This updates the new platform JSON logger to be ECS compliant. Internal representation of log fields are left as is, so there's a few translations. In some cases this may make sense for objects we define, e.g. pid, but others (Error) are more questionable. I'm open to replacing some of these.

Objects passed in the meta field are assigned to provide extended fields beyond message.

ecs reference
Closes #52226

pid -> process.pid
context -> log.logger
level -> log.level
error.stack -> error.stack_trace
error.name -> error.type

elasticmachine · 2020-07-08T18:28:51Z

Pinging @elastic/kibana-operations (Team:Operations)

src/core/server/logging/layouts/json_layout.ts

mshustov · 2020-07-09T15:46:40Z

src/core/server/logging/layouts/json_layout.ts

-      name: error.name,
-      stack: error.stack,
+      type: error.name,
+      stack_trace: error.stack,
    };
  }

  public format(record: LogRecord): string {
    return JSON.stringify({
      '@timestamp': moment(record.timestamp).format('YYYY-MM-DDTHH:mm:ss.SSSZ'),


could you add a test that @timestamp can be overridden? Should we @timestamp value customization for pattern layout as well?

e095437 for the test. 👍 it makes sense to be consistent on the merge behavior. I'm going to circle back on this one in just a moment

I took a closer look, the current behavior is open ended. I can replace @timestamp. The pattern layout has client-side type conversions like formats and timezone for the default fields, and the meta object is serialized to json.

merging the meta object into the record will throw a bunch of fields without formats at the logger. current behavior is json stringify, which will probably be strange for most ecs documents

overrides before or after the converstion layer.

I'm leaning towards addressing 1 separately and for 2overriding before - there aren't any yet but we don't want to miss any escape or string filters.

merging the meta object into the record will throw a bunch of fields without formats at the logger. current behavior is json stringify, which will probably be strange for most ecs documents

We stringify meta fields in pattern layout only, which is not subject to ECS format? However, I still don't see how users can reference data from the meta object without using meta. prefix.

I'm leaning towards addressing 1 separately and for 2overriding before - there aren't any yet but we don't want to miss any escape or string filters.

Are you going to create a separate issue?

src/core/server/logging/layouts/json_layout.ts

mshustov · 2020-07-09T16:06:20Z

src/core/server/logging/layouts/json_layout.ts

+      error: JsonLayout.errorToSerializableObject(record.error),
+      log: {
+        level: record.level.id.toUpperCase(),
+        logger: record.context,


How ES handles pattern layout for a given meta object? Does it enforces ECS compatible property name usage in conversion patterns? https://logging.apache.org/log4j/log4j-2.2/manual/layouts.html#PatternLayout

For now the field usage is best effort and under migration. They'll eventually move to prefixing custom fields with es.

https://github.com/elastic/elasticsearch/blob/master/test/logger-usage/src/main/java/org/elasticsearch/test/loggerusage/ESLoggerUsageChecker.java does some static checking, but there's no strict compliance yet

For now the field usage is best effort and under migration. They'll eventually move to prefixing custom fields with es.

What is our plan for this?

mshustov · 2020-07-09T16:13:00Z

src/core/server/logging/layouts/json_layout.ts

+      process: {
+        pid: record.pid,
+      },
+      ...record.meta,


Is it possible to provide typings for ECS-compatible meta object?

How detailed are you thinking? e.g. The whole spec, base objects, or defaults we're setting?

Might be worth digging into elastic/ecs#280

tylersmalley

Couple questions inline.

Another common property is the timings and request sizes. Those should be standardized as well.

src/core/server/logging/layouts/json_layout.test.ts

tylersmalley · 2020-07-14T14:23:44Z

src/core/server/logging/layouts/json_layout.ts

+            pid: record.pid,
+          },
+        },
+        record.meta


Similar to other comment, should we be placing record.meta under labels?

meta might be named incorrectly - i tried to leave internal terminology as intact as possible, but it's closer to fields or something. i could rename it

Could you create an issue? I'm fine to rename it.

jbudz · 2020-07-14T14:54:43Z

We do have timings with the timestamp field, or were you referring to something else? And then for request sizes - totally agree I just mostly punted on it for now, the organization is more at the architecture level. There's a couple other cases we want to cover - we will likely subclass JSON Logger with with http, process, ftr, optimizer, logs and so on. Audit logging will be the first tho

For a little context on the organization part - a few questions would be

field order for pattern input
sanitizing output (validation)
input validation
where the logger types live. does http live with the router or in the logger folder
type checking

jbudz · 2020-07-14T14:55:04Z

@elasticmachine merge upstream

jbudz · 2020-07-14T14:59:43Z

And then for the more specific logs - I'm thinking we'll want to provide the how and the structure, but mostly leave it up to the teams to decide. The surface area is quite large, we can certainly hook into some globals but there's other log levels.

tylersmalley

LGTM - I still see a lot of work to do in regards to ensuring alignment with ECS but feel that is blocked on us migrating to the new logger.

kibanamachine · 2020-07-14T16:36:44Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: a9a2fae

Build metrics

✅ unchanged

History

💚 Build #61018 succeeded 3916e3c
💔 Build #60895 failed c419f54
💚 Build #59910 succeeded 5e5c315
💔 Build #59881 failed e5a0992

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

mshustov

For a little context on the organization part - a few questions would be

field order for pattern input

How ES handles the order? I suspect they ignore fields not listed in a pattern layout.

sanitizing output (validation)

Do you mean adding a filtering mechanism? #57547

input validation

yeah, I thought about this, but considering the number of logs, it could affect the runtime perf. You can add this suggestion to #58261

where the logger types live. does http live with the router or in the logger folder
type checking

It seems like ECS dictates the format. Even in ES, they use a dedicated builder living in the Logger (in their case in AuditLogger https://github.com/elastic/elasticsearch/blob/aa0148735e3384b23d504d44ed4f2e20f64f1bca/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/audit/logfile/LoggingAuditTrail.java#L705)

mshustov · 2020-07-15T06:48:45Z

src/core/server/logging/layouts/json_layout.ts

-      pid: record.pid,
-    });
+    return JSON.stringify(
+      merge(


I'd rather we didn't use lodash.merge. It tripped us up several times with questionable functionality: for example, it merges array. BTW there is custom merge implementation in the repo: https://github.com/restrry/kibana/blob/a9a2fae0fc0e56ed3c7f0985e38e6436e72e160f/src/core/utils/merge.ts#L27

mshustov · 2020-07-15T06:55:26Z

src/core/server/logging/layouts/json_layout.ts

+    return JSON.stringify(
+      merge(
+        {
+          '@timestamp': moment(record.timestamp).format('YYYY-MM-DDTHH:mm:ss.SSSZ'),


formatting should be applied to the user @timestamp value as well. the same for error, level + tests required

mshustov · 2020-07-15T07:23:51Z

src/core/server/logging/layouts/json_layout.ts

+            pid: record.pid,
+          },
+        },
+        record.meta


Could you create an issue? I'm fine to rename it.

jbudz · 2020-07-15T13:43:38Z

Spoke with restry on slack - I'm going to merge this now for FF. The logger isn't in use yet, and there's one bug and two renames I'll address during freeze (and more probably)

I'm going to to recap and move^comments to the meta issue. I'll be following up with an implementation focusing on logging queries to elasticsearch.

@timestamp

* [logging] Format new platform json logging to ECS * update integration tests * merge instead of assign * add @timestamp override test * add partial merge test against log object * add object level override test * fix type error Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* master: [Form lib] Memoize form hook object and fix hook array deps (elastic#71237) [uiActions] Support emitting nested triggers and actions (elastic#70602) add short sleep before clicking Remove on sample data (elastic#71104) Fixed the beta badge layout. (elastic#71835) Restores task for downloading Chromium builds (elastic#71749) [logging] Format new platform json logging to ECS (elastic#71138) add policy details and update SO limit requests (elastic#71789) Convert vis_type_vega to Typescript (elastic#68915) [ML] Fix UI Actions context menu positioning for the Anomaly Swim Lane (elastic#71839)

kibanamachine · 2020-07-17T13:48:44Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71138 or prevent reminders by adding the backport:skip label.

kibanamachine · 2020-07-20T13:58:43Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71138 or prevent reminders by adding the backport:skip label.

kibanamachine · 2020-07-21T14:08:44Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 71138 or prevent reminders by adding the backport:skip label.

[logging] Format new platform json logging to ECS

e5a0992

jbudz added review Team:Operations Team label for Operations Team release_note:skip Skip the PR/issue when compiling release notes labels Jul 8, 2020

jbudz requested a review from mshustov July 8, 2020 18:28

jbudz requested a review from a team as a code owner July 8, 2020 18:28

jbudz added v7.9.0 v8.0.0 labels Jul 8, 2020

update integration tests

5e5c315

mshustov reviewed Jul 9, 2020

View reviewed changes

jbudz added 8 commits July 9, 2020 14:37

Merge branch 'master' into logging/ecs

6dffc2e

merge instead of assign

e3e7afd

add @timestamp override test

e095437

add partial merge test against log object

7fd0123

Merge branch 'master' into logging/ecs

62a9d1b

add object level override test

c419f54

Merge branch 'master' into logging/ecs

557b1c6

fix type error

3916e3c

tylersmalley suggested changes Jul 14, 2020

View reviewed changes

Merge branch 'master' into logging/ecs

a9a2fae

tylersmalley approved these changes Jul 14, 2020

View reviewed changes

mshustov reviewed Jul 15, 2020

View reviewed changes

jbudz merged commit 8bcecc0 into elastic:master Jul 15, 2020

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jul 17, 2020

mshustov mentioned this pull request Jul 19, 2020

[logging] Use Elastic Common Schema (ECS) #52226

Closed

jbudz added the backported label Jul 22, 2020

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Jul 22, 2020

mshustov mentioned this pull request Dec 18, 2020

[logging/json] use merge from kbn/std #86330

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[logging] Format new platform json logging to ECS #71138

[logging] Format new platform json logging to ECS #71138

jbudz commented Jul 8, 2020 •

edited by tylersmalley

Loading

elasticmachine commented Jul 8, 2020

mshustov Jul 9, 2020

jbudz Jul 13, 2020

jbudz Jul 13, 2020

mshustov Jul 15, 2020

mshustov Jul 9, 2020

jbudz Jul 13, 2020

mshustov Jul 15, 2020

mshustov Jul 9, 2020

jbudz Jul 13, 2020

tylersmalley left a comment

tylersmalley Jul 14, 2020

jbudz Jul 14, 2020

mshustov Jul 15, 2020

jbudz commented Jul 14, 2020 •

edited

Loading

jbudz commented Jul 14, 2020

jbudz commented Jul 14, 2020

tylersmalley left a comment

kibanamachine commented Jul 14, 2020

mshustov left a comment •

edited

Loading

mshustov Jul 15, 2020

mshustov Jul 15, 2020

mshustov Jul 15, 2020

jbudz commented Jul 15, 2020 •

edited

Loading

kibanamachine commented Jul 17, 2020

kibanamachine commented Jul 20, 2020

kibanamachine commented Jul 21, 2020

[logging] Format new platform json logging to ECS #71138

[logging] Format new platform json logging to ECS #71138

Conversation

jbudz commented Jul 8, 2020 • edited by tylersmalley Loading

elasticmachine commented Jul 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tylersmalley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbudz commented Jul 14, 2020 • edited Loading

jbudz commented Jul 14, 2020

jbudz commented Jul 14, 2020

tylersmalley left a comment

Choose a reason for hiding this comment

kibanamachine commented Jul 14, 2020

💚 Build Succeeded

Build metrics

History

mshustov left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbudz commented Jul 15, 2020 • edited Loading

kibanamachine commented Jul 17, 2020

kibanamachine commented Jul 20, 2020

kibanamachine commented Jul 21, 2020

jbudz commented Jul 8, 2020 •

edited by tylersmalley

Loading

jbudz commented Jul 14, 2020 •

edited

Loading

mshustov left a comment •

edited

Loading

jbudz commented Jul 15, 2020 •

edited

Loading