2022-07-25T08:57:46.801994014Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:57:46+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 10 in 64 seconds."}
2022-07-25T08:57:46.802307210Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:57:46+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64110ms."}
2022-07-25T08:58:50.878897083Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:58:50+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 11 in 64 seconds."}
2022-07-25T08:58:50.879293477Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:58:50+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64077ms."}
2022-07-25T08:59:54.953351424Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:59:54+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 12 in 64 seconds."}
2022-07-25T08:59:54.953831117Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:59:54+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64074ms."}
2022-07-25T09:00:59.034489002Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:00:59+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 13 in 64 seconds."}
2022-07-25T09:00:59.034889195Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:00:59+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64081ms."}
2022-07-25T09:02:03.101894263Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:02:03+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 14 in 64 seconds."}
2022-07-25T09:02:03.102293456Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:02:03+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64068ms."}
2022-07-25T09:03:07.190998929Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:03:07+00:00","tags":["error","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 15 in 64 seconds."}
2022-07-25T09:03:07.191358023Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:03:07+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64089ms."}
2022-07-25T09:04:11.286120578Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:04:11+00:00","tags":["info","savedobjects-service"],"pid":7,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> FATAL. took: 64095ms."}
2022-07-25T09:04:11.473282268Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:04:11+00:00","tags":["fatal","root"],"pid":7,"message":"Error: Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT step after 15 attempts, terminating.\n    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:144:29)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 1)\n    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:181:9)\n    at Server.start (/usr/share/kibana/src/core/server/server.js:330:31)\n    at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)\n    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)\n    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:229:5)"}
2022-07-25T09:04:11.591404242Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:04:11+00:00","tags":["info","plugins-system","standard"],"pid":7,"message":"Stopping all plugins."}
2022-07-25T09:04:11.592627921Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:04:11+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":7,"message":"Monitoring stats collection is stopped"}
2022-07-25T09:04:41.624120080Z lme_kibana.1.std1wbc037g1@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T09:04:41+00:00","tags":["warning","plugins-system","standard"],"pid":7,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}
2022-07-25T09:04:41.733692092Z lme_kibana.1.std1wbc037g1@vsmolme01    |
2022-07-25T09:04:41.733725092Z lme_kibana.1.std1wbc037g1@vsmolme01    |  FATAL  Error: Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT step after 15 attempts, terminating.
2022-07-25T09:04:41.733733092Z lme_kibana.1.std1wbc037g1@vsmolme01    |
2022-07-25T08:42:16.536806455Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:42:16+00:00","tags":["error","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: [no_shard_available_action_exception] Reason: [es01][10.0.0.4:9300][indices:data/read/open_reader_context]'. Retrying attempt 10 in 64 seconds."}
2022-07-25T08:42:16.537194425Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:42:16+00:00","tags":["info","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64160ms."}
2022-07-25T08:43:20.690968635Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:43:20+00:00","tags":["error","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: [no_shard_available_action_exception] Reason: [es01][10.0.0.4:9300][indices:data/read/open_reader_context]'. Retrying attempt 11 in 64 seconds."}
2022-07-25T08:43:20.691457607Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:43:20+00:00","tags":["info","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64154ms."}
2022-07-25T08:44:24.809502445Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:44:24+00:00","tags":["error","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: [no_shard_available_action_exception] Reason: [es01][10.0.0.4:9300][indices:data/read/open_reader_context]'. Retrying attempt 12 in 64 seconds."}
2022-07-25T08:44:24.853027515Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:44:24+00:00","tags":["info","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64118ms."}
2022-07-25T08:45:28.928344099Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:45:28+00:00","tags":["error","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 13 in 64 seconds."}
2022-07-25T08:45:28.928865281Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:45:28+00:00","tags":["info","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64119ms."}
2022-07-25T08:46:32.999237967Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:46:32+00:00","tags":["error","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 14 in 64 seconds."}
2022-07-25T08:46:32.999513059Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:46:32+00:00","tags":["info","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64071ms."}
2022-07-25T08:47:37.088888087Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:47:37+00:00","tags":["error","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] Action failed with 'search_phase_execution_exception: '. Retrying attempt 15 in 64 seconds."}
2022-07-25T08:47:37.089355175Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:47:37+00:00","tags":["info","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT. took: 64090ms."}
2022-07-25T08:48:41.164020254Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:48:41+00:00","tags":["info","savedobjects-service"],"pid":6,"message":"[.kibana_task_manager] OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT -> FATAL. took: 64075ms."}
2022-07-25T08:48:41.717530108Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:48:41+00:00","tags":["fatal","root"],"pid":6,"message":"Error: Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT step after 15 attempts, terminating.\n    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:144:29)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Promise.all (index 1)\n    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:181:9)\n    at Server.start (/usr/share/kibana/src/core/server/server.js:330:31)\n    at Root.start (/usr/share/kibana/src/core/server/root/index.js:69:14)\n    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:120:5)\n    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:229:5)"}
2022-07-25T08:48:41.931304057Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:48:41+00:00","tags":["info","plugins-system","standard"],"pid":6,"message":"Stopping all plugins."}
2022-07-25T08:48:41.932559266Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:48:41+00:00","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":6,"message":"Monitoring stats collection is stopped"}
2022-07-25T08:49:12.044523648Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    | {"type":"log","@timestamp":"2022-07-25T08:49:12+00:00","tags":["warning","plugins-system","standard"],"pid":6,"message":"\"eventLog\" plugin didn't stop in 30sec., move on to the next."}
2022-07-25T08:49:12.305878986Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    |
2022-07-25T08:49:12.305908586Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    |  FATAL  Error: Unable to complete saved object migrations for the [.kibana_task_manager] index: Unable to complete the OUTDATED_DOCUMENTS_SEARCH_OPEN_PIT step after 15 attempts, terminating.
2022-07-25T08:49:12.305920386Z lme_kibana.1.tfrhy29aj1f6@vsmolme01    |
lme@vsmolme01:/opt/lme/Chapter 3 Files$