Observability during deployment

[reyang]

Feature Request

The current demo seems to focus on observability when everything is already up and running.
I feel it might be important (and also cool) to be able to observe the system during setup, deployment or even decommission.

For example, maybe during the docker compose up, the 1st service to start is actually the UI (maybe Grafana or Kibana), which we can use to easily tell:

1. Which service is now up and running.
2. Any service is under deployment.
3. Any deployment issue.

Describe the solution you'd like:

Maybe:

1. Make sure the logging/metrics backend/UI are up and running.
2. Deploy and run other services.
3. Send deployment logs and service up/availability metrics during the deployment.

[wph95]

A possible scenario, using our helm-demo as an example

STEP

1. helm install otel-demo --step= bootstrap
   Install only our frontUI
   When you visit our frontUI, you can see that the other services are offline.

2. helm install otel-demo --step=monitor

   Install prometheus/grafana

3. helm install otel-demo --step=demo-app

   install our otel-demo
   While waiting, you can see the installation process through the frontUI admin page / grafana

Parts to discuss

1. step1 and step2 may be combined into one step.

2. How to gracefully turn on trace and log is something we can implement later.

3. It means that our helm chart has two modes.

   1. easy mode, install all services with one click
   2. step by step mode, you can enjoy Observability during deployment