[Observability Onboarding] Auto-detect flow #186106

thomheymann · 2024-06-12T16:10:14Z

Summary

Implements auto-detect based onboarding flow.
The new flow can only be accessed behind a hidden URL: /app/observabilityOnboarding/auto-detect?category=logs

Screenshots

Kibana

Terminal

…auto-detect-flow

obltmachine · 2024-06-12T16:10:28Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

thomheymann · 2024-06-12T16:10:28Z

/ci

…fix'

thomheymann · 2024-06-13T10:31:35Z

/ci

…ann/kibana into 183362-auto-detect-flow

…auto-detect-flow

thomheymann · 2024-06-15T21:17:23Z

/ci

…fix'

thomheymann · 2024-06-16T13:15:56Z

/ci

…auto-detect-flow

thomheymann · 2024-06-18T18:47:54Z

/ci

…fix'

thomheymann · 2024-06-19T06:56:59Z

/ci

…ann/kibana into 183362-auto-detect-flow

flash1293 · 2024-06-20T12:05:39Z

Other than that this looks pretty good to me - there are some components that can also be useful for the otel flow, but we can consolidate that after everything is merged

…ann/kibana into 183362-auto-detect-flow

…auto-detect-flow

thomheymann · 2024-06-20T15:50:21Z

This is the payload:

{"statusCode":400,"error":"Bad Request","message":"uri [/internal/observability_onboarding/flow/9f859b08-8212-445d-aeeb-b6540ebc9fc4/integrations/install] with method [post] exists but is not available with the current configuration"}

Maybe we need to change something for serverless in particular?

Ah, good catch. I think the curl command was missing some required internal request headers. I've added them in now.

…auto-detect-flow

flash1293

Really cool, left some high level comments

The install went through now, but the call to update the status to "complete" seems to have been run into a connection timeout:

Not sure whether that was a problem with our serverless setup or something inherent.

flash1293 · 2024-06-20T09:54:46Z

x-pack/plugins/observability_solution/observability_onboarding/public/assets/charts_screen.svg

Just a note - also added that as part of #183732

flash1293 · 2024-06-21T08:46:20Z

...servability_onboarding/public/application/quickstart_flows/auto_detect/auto_detect_panel.tsx

+                        title={i18n.translate(
+                          'xpack.observability_onboarding.autoDetectPanel.h3.getStartedWithNginxLabel',
+                          {
+                            defaultMessage: 'Get started with {pkgName} logs',


can we use the title of the integration instead of the technical name?

flash1293 · 2024-06-21T08:51:36Z

...servability_onboarding/public/application/quickstart_flows/auto_detect/auto_detect_panel.tsx

+                            <ul>
+                              {integration.kibanaAssets
+                                .filter((asset) => asset.type === 'dashboard')
+                                .map((dashboard) => (


Not all of them will be relevant (e.g. we don't support windows). I'm aware there is no easy way to tell which ones matter to the current user - if we can't come up with something I think it's not a big issue either

Yeh, we need a way of establishing which dashboard is the main one we want to present. Usually there's an "overview" dashboard that can be considered the best entry point so we might be able to do a naive pattern matching.

Ideally there would also be a way of determining whether a dashboard is relevant in the first place (e.g. whether it contains any data) but I'm not sure this can be easily done currently.

I think this panel needs another design iteration where we look at what information is returned from fleet for each of the supported integrations and then how we want to present that.

Makes sense, could you prepare that discussion for our next sync?

I agree we should have some logic on which dashboard should be the one we want to show and group the rest on the link to see all of them. As mentioned, usually there is an Overview. In this case we'll need to evaluate host vs system metrics overview dashboards.

flash1293 · 2024-06-21T08:53:01Z

...servability_onboarding/public/application/quickstart_flows/auto_detect/auto_detect_panel.tsx

+                                    <LocatorButtonEmpty<DashboardLocatorParams>
+                                      locator={DASHBOARD_APP_LOCATOR}
+                                      params={{ dashboardId: dashboard.id }}
+                                      target="_blank"


I think it's great opening in a new tab! However, when using it I didn't trust it because there's no indication this will happen, should we append a suitable icon to the end?

Adds some visual noise as well though, so not 100% sure

Unfortunately, buttons can only have one icon. It think it would be too confusing otherwise.

We could change the icon to an external link icon on the right hand side but then we would have to remove the current icons

Let's discuss with @sileschristian once he's back - no strong opinion on it

@thomheymann

We could change the icon to an external link icon on the right hand side but then we would have to remove the current icons

This makes sense. Let's do this. Dashboard icon is not bringing that much of value. Also, we are missing a copy that we usually we had before the link that gives better context.

mykolaharmash

Looks good, just a few minor comments

...lity_onboarding/public/application/quickstart_flows/auto_detect/copy_to_clipboard_button.tsx

...on/observability_onboarding/public/application/quickstart_flows/auto_detect/empty_prompt.tsx

mykolaharmash · 2024-06-21T10:17:52Z

...rvability_onboarding/public/application/quickstart_flows/auto_detect/accordion_with_icon.tsx

+            <EuiIcon type={iconType} size="l" />
+          </EuiFlexItem>
+          <EuiFlexItem>
+            <EuiTitle size="xs" css={rest.isDisabled ? { color: 'inherit' } : undefined}>


nit: better to have isDisabled as a de-structured prop as you use it directly

But then I would also have to manually pass it into accordion. Leaving it inside rest means it will get spread with the rest of the props that are just pass through props from the EuiAccordion.

...ervability_onboarding/public/application/quickstart_flows/auto_detect/progress_indicator.tsx

mykolaharmash · 2024-06-21T10:43:58Z

...rvability_onboarding/public/application/quickstart_flows/auto_detect/use_onboarding_flow.tsx

+  const assetsState = useAsync(async () => {
+    if (installedIntegrations.length === 0) {
+      return [];
+    }
+    const assetsMetadata = await fleet.hooks.epm.getBulkAssets({
+      assetIds: installedIntegrations
+        .map((integration) => integration.kibanaAssets)
+        .flat() as AssetSOObject[],
+    });
+    return installedIntegrations.map((integration) => {
+      return {
+        ...integration,
+        // Enrich installed Kibana assets with metadata from Fleet API (e.g. title, description, etc.)
+        kibanaAssets: integration.kibanaAssets.reduce<GetBulkAssetsResponse['items']>(
+          (acc, asset) => {
+            const assetWithMetadata = assetsMetadata.data?.items.find(({ id }) => id === asset.id);
+            if (assetWithMetadata) {
+              acc.push(assetWithMetadata);
+            }
+            return acc;
+          },
+          []
+        ),
+      };
+    });
+  }, [installedIntegrations.length]); // eslint-disable-line react-hooks/exhaustive-deps


Just a personal preference, no need for changes, but I'd keep metadata as a separate structure and let each installed integration look up stuff there while rendering.

The drawback of that approach would be that you'd be making multiple unecessary HTTP requests for each integration separately to load that metadata in. Happy to refactor at a later stage if this causes issues but gonna keep as is for now

flash1293 · 2024-06-21T11:25:41Z

I guess you planned to do this anyway, but happy to merge it in in a hidden way so testing becomes easier and then do the polishing in a follow-up PR.

thomheymann · 2024-06-21T13:42:06Z

I guess you planned to do this anyway, but happy to merge it in in a hidden way so testing becomes easier and then do the polishing in a follow-up PR.

I have already removed the auto detection card so the flow is only accessible via the hidden URL

kibana-ci · 2024-06-24T09:20:18Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 8464513
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-186106-846451358406
Observability Deployment

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`observabilityOnboarding`	201	215	+14

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`observabilityOnboarding`	200.2KB	209.1KB	+8.9KB

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`observabilityOnboarding`	14	15	+1

References to deprecated APIs

id	before	after	diff
`observabilityOnboarding`	1	3	+2

Total ESLint disabled count

id	before	after	diff
`observabilityOnboarding`	18	19	+1

History

💛 Build #217167 was flaky 17c73f7
💛 Build #217092 was flaky 60b21ff
💛 Build #216953 was flaky 6537d89
💛 Build #216853 was flaky 884fda5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Depends on: elastic/elastic-agent#4754 Depends on: #186106 Closes: #182407 ## Summary Adds a Kubernetes onboarding quick start flow using `kubectl kustomize` command. ![CleanShot 2024-06-18 at 15 10 27@2x](https://github.com/elastic/kibana/assets/793851/522d2481-6a0e-43d3-b9ef-d09ee9953b3c) ## How to test 1. Run Kibana and ES locally (make sure to expose ES on 0.0.0.0 so elastic agent can reach it from within a container, I use this command `yarn es snapshot --license trial -E xpack.security.authc.api_key.enabled=true -E http.host=0.0.0.0`) 2. Setup a test cluster with [minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Fx86-64%2Fstable%2Fbinary+download) 3. Open Kibana and navigate to the Onboarding screen 4. Make sure Kubernetes quick start card is visible under the infrastructure category and click on it 5. Copy the command snippet 6. Paste the command into a terminal, but don't run it yet 7. Replace `localhost` in the command with you local IP `ipconfig getifaddr en0` 8. In case elastic/elastic-agent#4754 was not merged yet, you'd need to also clone the elastic-agent repo and replace the template URL with a local path to the `elastic-agent-kustomize/default/elastic-agent-standalone` folder. 9. Run the command and make sure all resources were created 10. Go back to Kibana, after ~1 minute UI should identify that the data was ingested 11. Click on the cluster overview link and make sure it works --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Depends on: elastic/elastic-agent#4754 Depends on: elastic#186106 Closes: elastic#182407 ## Summary Adds a Kubernetes onboarding quick start flow using `kubectl kustomize` command. ![CleanShot 2024-06-18 at 15 10 27@2x](https://github.com/elastic/kibana/assets/793851/522d2481-6a0e-43d3-b9ef-d09ee9953b3c) ## How to test 1. Run Kibana and ES locally (make sure to expose ES on 0.0.0.0 so elastic agent can reach it from within a container, I use this command `yarn es snapshot --license trial -E xpack.security.authc.api_key.enabled=true -E http.host=0.0.0.0`) 2. Setup a test cluster with [minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Fx86-64%2Fstable%2Fbinary+download) 3. Open Kibana and navigate to the Onboarding screen 4. Make sure Kubernetes quick start card is visible under the infrastructure category and click on it 5. Copy the command snippet 6. Paste the command into a terminal, but don't run it yet 7. Replace `localhost` in the command with you local IP `ipconfig getifaddr en0` 8. In case elastic/elastic-agent#4754 was not merged yet, you'd need to also clone the elastic-agent repo and replace the template URL with a local path to the `elastic-agent-kustomize/default/elastic-agent-standalone` folder. 9. Run the command and make sure all resources were created 10. Go back to Kibana, after ~1 minute UI should identify that the data was ingested 11. Click on the cluster overview link and make sure it works --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit 141e619)

Depends on: elastic/elastic-agent#4754 Depends on: elastic#186106 Closes: elastic#182407 ## Summary Adds a Kubernetes onboarding quick start flow using `kubectl kustomize` command. ![CleanShot 2024-06-18 at 15 10 27@2x](https://github.com/elastic/kibana/assets/793851/522d2481-6a0e-43d3-b9ef-d09ee9953b3c) ## How to test 1. Run Kibana and ES locally (make sure to expose ES on 0.0.0.0 so elastic agent can reach it from within a container, I use this command `yarn es snapshot --license trial -E xpack.security.authc.api_key.enabled=true -E http.host=0.0.0.0`) 2. Setup a test cluster with [minikube](https://minikube.sigs.k8s.io/docs/start/?arch=%2Fmacos%2Fx86-64%2Fstable%2Fbinary+download) 3. Open Kibana and navigate to the Onboarding screen 4. Make sure Kubernetes quick start card is visible under the infrastructure category and click on it 5. Copy the command snippet 6. Paste the command into a terminal, but don't run it yet 7. Replace `localhost` in the command with you local IP `ipconfig getifaddr en0` 8. In case elastic/elastic-agent#4754 was not merged yet, you'd need to also clone the elastic-agent repo and replace the template URL with a local path to the `elastic-agent-kustomize/default/elastic-agent-standalone` folder. 9. Run the command and make sure all resources were created 10. Go back to Kibana, after ~1 minute UI should identify that the data was ingested 11. Click on the cluster overview link and make sure it works --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

amolnater-qasource · 2024-07-17T07:04:26Z

Hi @gbamparop

We have revalidated this script on 8.15.0 BC2 kibana cloud environment available by adding /app/observabilityOnboarding/auto-detect?category=logs to the URL.

Build details:

VERSION: 8.15.0 BC2
BUILD: 76111
COMMIT: 8d6510f30e7a52246ccebbf7d913a59b498c3bc2

Platforms:

Ubuntu 24.04
SLES 15
MAC Sonoma 14.5

Observations:

Steps gets updated in the CLI on running the available command and similarly the flow gets completed under UI.
We are successfully able to run the script on Linux and Mac platforms.
Data is generated for the installed agent and Dashboards are created as per the integrations.
Further, currently the script isn't testable on Windows due to && sudo in the command.

SLES15:

Ubuntu 24.04:

MAC Sonoma 14.5:

Elastic - Google Chrome 2024-07-17 11-56-30.zip

Error on Windows:

Please let us know if we are missing anything here.

Thanks!!

flash1293 · 2024-07-17T10:00:08Z

Thanks for testing @amolnater-qasource !

The windows case is expected for now - could you test with apache, nginx and docker being installed on mac and linux as well to make sure the detection and data collection works as expected for these as well?

amolnater-qasource · 2024-07-17T12:17:06Z

Hi @flash1293

Thank you for the update.

We have installed apache, nginx and docker on the linux and mac. Please find below detailed observations.

Observations:

Linux:

Able to view dashboards for all the 3 installations with linux host data present in it.

Screen Recording:

Elastic.-.Google.Chrome.2024-07-17.15-54-27.mp4

MAC:

Further on mac when we have installed the agent no dashboards for any of these services are available on UI.

Screenshot:

Please let us know if we are missing anything here.

Thanks!!

flash1293 · 2024-07-17T13:06:53Z

Thanks for checking @amolnater-qasource - I think the detection we built for nginx, apache and docker doesn't work for the way these are installed on mac, I will check back with the team.

thomheymann · 2024-07-26T08:55:44Z

@flash1293 The paths we use in the script are based on the ones provided by EPM. For docker that is:

  - id: filestream-docker
    type: filestream
    streams:
      - id: 'docker-container-logs-${docker.container.name}-${docker.container.id}'
        data_stream:
          dataset: docker.container_logs
          type: logs
          elasticsearch:
            dynamic_dataset: true
            dynamic_namespace: true
        paths:
          - '/var/lib/docker/containers/${docker.container.id}/*-json.log'
        parsers:
          - container:
              stream: all
              format: docker

It looks like Docker desktop on macOS does not use the same paths.
This would be an issue with the existing Elastic Agent / Fleet flow as well but I guess it's made worse in our case since we're claiming that we can auto-detect these apps. I wonder if we need to add additional known log file paths or if we should be smarter with the detection and look for running processes or knowns commands in the PATH.

flash1293 · 2024-07-26T10:29:24Z

This would be an issue with the existing Elastic Agent / Fleet flow as well but I guess it's made worse in our case since we're claiming that we can auto-detect these apps.

Right, the claim right above is the main concern - it's just something the integration can't do today.

I wonder if we need to add additional known log file paths or if we should be smarter with the detection and look for running processes or knowns commands in the PATH.

In general, in docker desktop the log files aren't even in the file system of the host by default, so it's not easily possible to collect them in the first place. In theory it's possible to get the logs via the /var/run/docker.sock socket instead of consuming files, but that would be a much larger change to the docker integration - if that would be available, we could detect docker by the presence of the socket instead of looking for certain files. Right now, we could do this detection strategy too, but then it would only get the docker metrics (which are collected through the socket), not the docker logs.

thomheymann added 4 commits June 12, 2024 11:00

Add auto-detect onboarding flow

63e1321

Merge branch 'main' of https://github.com/elastic/kibana into 183362-…

ee37a72

…auto-detect-flow

Parallelize installation and UI fixes

d30158c

Fix accordion id

6524770

thomheymann added release_note:skip Skip the PR/issue when compiling release notes Feature: Observability Onboarding labels Jun 12, 2024

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

3c36774

…fix'

thomheymann added 4 commits June 13, 2024 12:30

Refactor copy to clipboard

4fb7baf

Merge branch '183362-auto-detect-flow' of https://github.com/thomheym…

22fa187

…ann/kibana into 183362-auto-detect-flow

Fix API key creation and open links in new tab

4bd2fc5

Merge branch 'main' of https://github.com/elastic/kibana into 183362-…

09a92e9

…auto-detect-flow

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

37dff1a

…fix'

mykolaharmash mentioned this pull request Jun 18, 2024

[Onboarding] k8s quickstart flow #186380

Merged

thomheymann added 2 commits June 18, 2024 19:43

Add links to dashboards

b15fbdc

Merge branch 'main' of https://github.com/elastic/kibana into 183362-…

cc18fed

…auto-detect-flow

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

1703e14

…fix'

thomheymann marked this pull request as ready for review June 18, 2024 20:33

thomheymann requested review from a team as code owners June 18, 2024 20:33

thomheymann added 2 commits June 19, 2024 08:26

Use EUI copy component

6c76c59

Merge branch '183362-auto-detect-flow' of https://github.com/thomheym…

3c58df5

…ann/kibana into 183362-auto-detect-flow

botelastic bot added the ci:project-deploy-observability Create an Observability project label Jun 19, 2024

thomheymann added 3 commits June 20, 2024 16:46

Add elastic internal origin header

0d7b6ca

Merge branch '183362-auto-detect-flow' of https://github.com/thomheym…

d48d2f8

…ann/kibana into 183362-auto-detect-flow

Merge branch 'main' of https://github.com/elastic/kibana into 183362-…

6537d89

…auto-detect-flow

thomheymann added 2 commits June 21, 2024 09:28

Hide auto-detect flow

6fdafa7

Merge branch 'main' of https://github.com/elastic/kibana into 183362-…

60b21ff

…auto-detect-flow

flash1293 reviewed Jun 21, 2024

View reviewed changes

Add title

a085986

mykolaharmash approved these changes Jun 21, 2024

View reviewed changes

Move components to shared

17c73f7

pgayvallet approved these changes Jun 24, 2024

View reviewed changes

Merge branch 'main' into 183362-auto-detect-flow

8464513

thomheymann merged commit f015066 into elastic:main Jun 24, 2024
24 checks passed

kibanamachine added v8.15.0 backport:skip This commit does not require backporting labels Jun 24, 2024

flash1293 mentioned this pull request Jul 25, 2024

Add hosts/auto-detection + K8s quick start plus overview elastic/observability-docs#4073

Merged

8 tasks

thomheymann mentioned this pull request Aug 20, 2024

[Observability Onboarding] Add placeholders and comments for variables without default values #190794

Closed

[Observability Onboarding] Auto-detect flow #186106

[Observability Onboarding] Auto-detect flow #186106

Conversation

thomheymann commented Jun 12, 2024 • edited Loading

Summary

Screenshots

Kibana

Terminal

obltmachine commented Jun 12, 2024

🤖 GitHub comments

thomheymann commented Jun 12, 2024

thomheymann commented Jun 13, 2024

thomheymann commented Jun 15, 2024

thomheymann commented Jun 16, 2024

thomheymann commented Jun 18, 2024

thomheymann commented Jun 19, 2024

flash1293 commented Jun 20, 2024

thomheymann commented Jun 20, 2024

flash1293 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mykolaharmash left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flash1293 commented Jun 21, 2024

thomheymann commented Jun 21, 2024

kibana-ci commented Jun 24, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Module Count

Async chunks

ESLint disabled line counts

References to deprecated APIs

Total ESLint disabled count

History

amolnater-qasource commented Jul 17, 2024

flash1293 commented Jul 17, 2024

amolnater-qasource commented Jul 17, 2024 • edited Loading

Linux:

MAC:

flash1293 commented Jul 17, 2024 • edited Loading

thomheymann commented Jul 26, 2024

flash1293 commented Jul 26, 2024 • edited Loading

thomheymann commented Jun 12, 2024 •

edited

Loading

kibana-ci commented Jun 24, 2024 •

edited

Loading

amolnater-qasource commented Jul 17, 2024 •

edited

Loading

flash1293 commented Jul 17, 2024 •

edited

Loading

flash1293 commented Jul 26, 2024 •

edited

Loading