[core.logging] Update ECS typings to 8.0.0

[lukeelmers]

In Core, we maintain a complete set of typings for ECS to enforce that any LogMeta added by plugins is ECS-compliant:

kibana/packages/kbn-logging/src/ecs/index.ts

Lines 57 to 97 in 12b245c

	/**
	* Represents the full ECS schema.
	*
	* @public
	*/
	export type Ecs = EcsBase &
	EcsTracing & {
	ecs: EcsField;
	agent?: EcsAgent;
	as?: EcsAutonomousSystem;
	client?: EcsClient;
	cloud?: EcsCloud;
	container?: EcsContainer;
	destination?: EcsDestination;
	dns?: EcsDns;
	error?: EcsError;
	event?: EcsEvent;
	file?: EcsFile;
	group?: EcsGroup;
	host?: EcsHost;
	http?: EcsHttp;
	log?: EcsLog;
	network?: EcsNetwork;
	observer?: EcsObserver;
	organization?: EcsOrganization;
	package?: EcsPackage;
	process?: EcsProcess;
	registry?: EcsRegistry;
	related?: EcsRelated;
	rule?: EcsRule;
	server?: EcsServer;
	service?: EcsService;
	source?: EcsSource;
	threat?: EcsThreat;
	tls?: EcsTls;
	url?: EcsUrl;
	user?: EcsUser;
	user_agent?: EcsUserAgent;
	vulnerability?: EcsVulnerability;
	};

ECS is working on finalizing their list of breaking changes for 8.0. We will need to update our types accordingly.

Note that we are also currently on ECS 1.9.0, so we'll also need to go through and make sure we've included any updates from 1.10 and 1.11:

• https://github.com/elastic/ecs/releases/tag/v1.10.0
• https://github.com/elastic/ecs/releases/tag/v1.11.0

(Someday, it would be nice to contribute to ECS by adding a generator for TypeScript, which would make this process a bit easier)

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[LeeDr]

I've been spending time over the last few days trying to ingest kibana logs (and audit logs) with filebeat and struggling with a few items. They'll probably be resolved in this issue but I'll mention them here anyway.

1. For the audit.log there's a mismatch between the Kibana and filebeat Kibana module. Our docs have an example with
   fileName: ./audit.log. But the filebeat kibana module (at least for an RPM on CentOS) is looking for
   Configured paths: [/var/log/kibana/*_audit.json]
2. at least for an RPM install where Kibana runs as a service, using a relative path for that audit log filename caused Kibana to crash. I had to give it a full path like /var/log/kibana/kibana_audit.json
3. Prior to the current build of master, the Kibana (non-ECS) logs had separate fields for method, statusCode, req, etc. But in the current master all of that is in message which means I have to parse it out using a runtime field. I think these are all going to be added into the ECS logging.
4. beats tests for the kibana module seem to be using really old test data from version 6.3 of the stack so we need to coordinate updates to the kibana module and the tests to get them up to date. https://github.com/elastic/beats/blob/master/filebeat/module/kibana/log/test/test.log

[mshustov]

1. For the audit.log there's a mismatch between the Kibana and filebeat Kibana module. Our docs have an example with
   fileName: ./audit.log. But the filebeat kibana module (at least for an RPM on CentOS) is looking for
   Configured paths: [/var/log/kibana/*_audit.json]
2. at least for an RPM install where Kibana runs as a service, using a relative path for that audit log filename caused Kibana to crash. I had to give it a full path like /var/log/kibana/kibana_audit.json

cc @elastic/kibana-security

3. Prior to the current build of master, the Kibana (non-ECS) logs had separate fields for method, statusCode, req, etc. But in the current master all of that is in message which means I have to parse it out using a runtime field. I think these are all going to be added into the ECS logging.

Yes, these are added to ECS-JSON logs.
response code as http.response.status_code
method as http.request.method

Not sure what req is. Could you elaborate? @LeeDr
The full example of ECS-JSON log record for http.response is:

{
   "client":{
      "ip":"127.0.0.1"
   },
   "http":{
      "request":{
         "method":"GET",
         "mime_type":null,
         "referrer":"http://localhost:5601/login?next=%2Fbon%2F",
         "headers":{
            "connection":"keep-alive",
            "sec-ch-ua":"\"Chromium\";v=\"94\", \"Google Chrome\";v=\"94\", \";Not A Brand\";v=\"99\"",
            "dnt":"1",
            "sec-ch-ua-mobile":"?0",
            "user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36",
            "sec-ch-ua-platform":"\"macOS\"",
            "accept":"*/*",
            "sec-fetch-site":"same-origin",
            "sec-fetch-mode":"no-cors",
            "sec-fetch-dest":"script",
            "referer":"http://localhost:5601/login?next=%2Fbon%2F",
            "accept-encoding":"gzip, deflate, br",
            "accept-language":"en-GB,en;q=0.9,en-US;q=0.8,ru;q=0.7,de;q=0.6,fr;q=0.5",
            "cookie":"[REDACTED]",
            "x-forwarded-for":"127.0.0.1",
            "x-forwarded-port":"59814",
            "x-forwarded-proto":"http",
            "x-forwarded-host":"localhost:5601",
            "host":"localhost:5601",
            "traceparent":"00-129c4dc90acf028fc277593883b5200f-0c20a773443c7de8-01",
            "tracestate":"es=s:1",
            "elastic-apm-traceparent":"00-129c4dc90acf028fc277593883b5200f-0c20a773443c7de8-01"
         }
      },
      "response":{
         "body":{
            "bytes":36880
         },
         "status_code":200,
         "headers":{
            "etag":"\"740c15b82af97f593e63c68833ae7c68237c3ab4-/bon/9007199254740991/bundles/plugin/expressionRevealImage/1.0.0/-gzip\"",
            "cache-control":"must-revalidate",
            "content-type":"application/javascript; charset=utf-8",
            "x-content-type-options":"nosniff",
            "referrer-policy":"no-referrer-when-downgrade",
            "kbn-name":"Mikhails-MBP",
            "kbn-license-sig":"092c91df379ac95bfcefcf05c5937d788dbbd74fd785706669676462d66ad323",
            "vary":"accept-encoding",
            "content-encoding":"gzip"
         },
         "responseTime":25
      }
   },
   "url":{
      "path":"/9007199254740991/bundles/plugin/expressionRevealImage/1.0.0/expressionRevealImage.plugin.js",
      "query":""
   },
   "user_agent":{
      "original":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"
   },
   "ecs":{
      "version":"1.9.0"
   },
   "@timestamp":"2021-10-01T12:03:28.459+02:00",
   "message":"GET /9007199254740991/bundles/plugin/expressionRevealImage/1.0.0/expressionRevealImage.plugin.js 200 25ms - 36.0KB",
   "log":{
      "level":"DEBUG",
      "logger":"http.server.response"
   },
   "process":{
      "pid":86350
   }
}

4. beats tests for the kibana module seem to be using really old test data from version 6.3 of the stack so we need to coordinate updates to the kibana module and the tests to get them up to date.

Would you mind creating an issue in beats repo and tag the Kibana team?

[lukeelmers]

beats tests for the kibana module seem to be using really old test data from version 6.3 of the stack so we need to coordinate updates to the kibana module and the tests to get them up to date.

Would you mind creating an issue in beats repo and tag the Kibana team?

@LeeDr Alternatively you could post a note to the beats issue that's tracking the 8.0 upgrade of the kibana filebeat module: elastic/beats#24136