component: Add linkerd #690

surajssd · 2020-07-01T12:51:51Z

To test manually create following config in your lokocfg file:

component "experimental-linkerd" {}

TODOs:

~~This installs lot of stuff like grafana, prometheus which can be disabled and settings done so that we can use prometheus operator.~~ It is not possible right now to decouple the prometheus shipped with linkerd but there is ongoing work on this: Move Prometheus as an Add-On linkerd/linkerd2#4362 Once that is merged and released we can think about it.
Use the values-ha.yaml compared to the normal values.yaml file.
I think the way currently certs are generated on every apply new certs will be created, see if it affects the general workings of the workloads.
~~For testing random patch/dev versions we need to have a way to specify which version of linkerd to deploy.~~

Tests:

Add tests to verify things with and without linkerd installation.

invidian · 2020-07-03T11:31:32Z

It seems the tests are failing.

surajssd · 2020-07-03T11:37:25Z

It seems the tests are failing.

My hypothesis is that linkerd installation is not complete and we try to create pods and they fail on some webhook validation. There is no retry logic in pod creation.

assets/components/linkerd2/values.yaml

pkg/components/linkerd/manifest.go

test/components/linkerd/linkerd_test.go

ci/aks/aks-cluster.lokocfg.envsubst

pkg/components/linkerd/component.go

surajssd · 2020-07-03T13:12:20Z

Now CI tasks fail with following error:

+ echo 'Sleeping for 30 seconds. Waiting for external-dns to clear DNS records.'
Sleeping for 30 seconds. Waiting for external-dns to clear DNS records.
+ sleep 30
+ '[' 2 = 0 ']'

iaguis · 2020-07-03T13:20:51Z

The errors seem to be further up:

--- FAIL: TestNoExtraSSHKeysOnNodes (300.11s)

--- FAIL: TestFileCreatedByCLCSnippetExistsOnNodes (300.11s)

surajssd · 2020-07-07T12:05:03Z

On the CI the linkerd installation is stuck and following happens:

Events from the linkerd namespace:

linkerd       14m         Warning   FailedCreate              replicaset/linkerd-controller-7c7d9bf56              Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s: dial tcp 10.3.192.109:443: connect: connection refused
linkerd       36m         Normal    ScalingReplicaSet         deployment/linkerd-controller                        Scaled up replica set linkerd-controller-7c7d9bf56 to 2
linkerd       14m         Warning   FailedCreate              replicaset/linkerd-destination-84b94d4df5            Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s: dial tcp 10.3.192.109:443: connect: connection refused
linkerd       36m         Normal    ScalingReplicaSet         deployment/linkerd-destination                       Scaled up replica set linkerd-destination-84b94d4df5 to 2
linkerd       14m         Warning   FailedCreate              replicaset/linkerd-grafana-8648658f4b                Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s: dial tcp 10.3.192.109:443: connect: connection refused
linkerd       36m         Normal    ScalingReplicaSet         deployment/linkerd-grafana                           Scaled up replica set linkerd-grafana-8648658f4b to 1
linkerd       14m         Warning   FailedCreate              replicaset/linkerd-identity-6c7b88bcf8               Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s: dial tcp 10.3.192.109:443: connect: connection refused
linkerd       36m         Normal    ScalingReplicaSet         deployment/linkerd-identity                          Scaled up replica set linkerd-identity-6c7b88bcf8 to 2
linkerd       14m         Warning   FailedCreate              replicaset/linkerd-prometheus-748b48c8c9             Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s: dial tcp 10.3.192.109:443: connect: connection refused
linkerd       36m         Normal    ScalingReplicaSet         deployment/linkerd-prometheus                        Scaled up replica set linkerd-prometheus-748b48c8c9 to 1
linkerd       14m         Warning   FailedCreate              replicaset/linkerd-proxy-injector-6b979484f          Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s: dial tcp 10.3.192.109:443: connect: connection refused
linkerd       36m         Normal    ScalingReplicaSet         deployment/linkerd-proxy-injector                    Scaled up replica set linkerd-proxy-injector-6b979484f to 2
linkerd       14m         Warning   FailedCreate              replicaset/linkerd-sp-validator-6896bc59c4           Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s: dial tcp 10.3.192.109:443: connect: connection refused
linkerd       36m         Normal    ScalingReplicaSet         deployment/linkerd-sp-validator                      Scaled up replica set linkerd-sp-validator-6896bc59c4 to 2
linkerd       14m         Warning   FailedCreate              replicaset/linkerd-tap-8f79d6c5c                     Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s: dial tcp 10.3.192.109:443: connect: connection refused
linkerd       36m         Normal    ScalingReplicaSet         deployment/linkerd-tap                               Scaled up replica set linkerd-tap-8f79d6c5c to 2
linkerd       14m         Warning   FailedCreate              replicaset/linkerd-web-8558657b69                    Error creating: Internal error occurred: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s: dial tcp 10.3.192.109:443: connect: connection refused
linkerd       36m         Normal    ScalingReplicaSet         deployment/linkerd-web                               Scaled up replica set linkerd-web-8558657b6

These are the apiserver errors:

I0707 11:56:57.236263       1 trace.go:116] Trace[1183028621]: "Create" url:/api/v1/namespaces/linkerd/pods,user-agent:kube-controller-manager/v1.18.3 (linux/amd64) kubernetes/2e7996e/system:serviceaccount:kube-system:replicaset-controller,client:10.0.4.35 (started: 2020-07-07 11:56:56.213473697 +0000 UTC m=+2383.280605006) (total time: 1.02276867s):
I0707 11:56:57.363868       1 trace.go:116] Trace[1840209148]: "Call mutating webhook" configuration:linkerd-proxy-injector-webhook-config,webhook:linkerd-proxy-injector.linkerd.io,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:301278b9-e5bc-411b-a77f-46c1718a9a51 (started: 2020-07-07 11:56:56.344687474 +0000 UTC m=+2383.411818776) (total time: 1.019130746s):
W0707 11:56:57.363946       1 dispatcher.go:181] Failed calling webhook, failing closed linkerd-proxy-injector.linkerd.io: failed calling webhook "linkerd-proxy-injector.linkerd.io": Post https://linkerd-proxy-injector.linkerd.svc:443/?timeout=30s: dial tcp 10.3.192.109:443: connect: connection refused

surajssd · 2020-07-07T13:35:13Z

I was able to recreate this error on HA AWS locally. It is not a problem on any form of Packet.

surajssd · 2020-07-07T14:43:49Z

Now it fails with following error:

        linkerd-existence
        -----------------
        √ 'linkerd-config' config map exists
        √ heartbeat ServiceAccount exist
        √ control plane replica sets are ready
        × no unschedulable pods
            linkerd-identity-76dd9cc969-9tdt4: 0/4 nodes are available: 2 Insufficient cpu, 2 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
            see https://linkerd.io/checks/#l5d-existence-unschedulable-pods for hints

invidian

Just some nits, otherwise LGTM

pkg/components/linkerd/component.go

surajssd · 2020-08-25T11:08:13Z

PTAL: @iaguis

invidian

LGTM, except I'd still change generateCertificates to not have a receiver and simply returns cert {} struct.

pkg/components/linkerd/component.go

This commit adds assets for linkerd2 chart at version `stable-2.8.1`. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

- Use cert creation API from linkerd. - Vendor `linkerd` libraries needed for cert generation. - Add content from `values-ha.yaml` file to the `manifest.go` to add support for HA installation. In the HA mode user can specify the number of controllers that should be created using a variable `controller_replicas`. **Caution**: If the number of linkerd controller count is more than the available worker nodes then some pods may stay in `Pending` state. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

In AWS install non-HA setup. CI on AWS currently does not have enough resources to run multiple replicas of the linkerd pods. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

- Add test to verify if the deployments are up and running. - Add test to run `linkerd check` against the apiserver. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

If pod creation fails for some reason in first attempt then try again. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

- Adds ServiceMonitors to monitor linkerd control plane pods. - Adds entry in the components prometheus metrics list to test if the linkerd metrics are being scraped. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

knrt10

Looks great @surajssd 🎉 .

surajssd · 2020-08-27T06:57:00Z

Tests pass 🎉 Thanks for your patient review @invidian and @knrt10.

surajssd force-pushed the surajssd/add-linkerd2 branch 5 times, most recently from c55f778 to 74c2647 Compare July 3, 2020 08:49

surajssd marked this pull request as ready for review July 3, 2020 08:52

surajssd requested review from invidian, ipochi, johananl and pothos as code owners July 3, 2020 08:52

invidian suggested changes Jul 3, 2020

View reviewed changes

surajssd force-pushed the surajssd/add-linkerd2 branch 7 times, most recently from b5b7cf9 to 53fb95e Compare July 7, 2020 09:33

surajssd force-pushed the surajssd/add-linkerd2 branch from 53fb95e to c830dfd Compare July 7, 2020 14:18

surajssd mentioned this pull request Jul 10, 2020

Discrepancy in lokoctl installation and helm installation #711

Closed

surajssd force-pushed the surajssd/add-linkerd2 branch 3 times, most recently from 3fece05 to ed9ff4a Compare July 10, 2020 10:37

invidian suggested changes Aug 18, 2020

View reviewed changes

pkg/components/linkerd/component.go Outdated Show resolved Hide resolved

pkg/components/linkerd/component.go Outdated Show resolved Hide resolved

pkg/components/linkerd/component.go Outdated Show resolved Hide resolved

pkg/components/linkerd/component.go Outdated Show resolved Hide resolved

surajssd force-pushed the surajssd/add-linkerd2 branch 2 times, most recently from 79f9f82 to 1baa756 Compare August 19, 2020 12:20

invidian suggested changes Aug 19, 2020

View reviewed changes

pkg/components/linkerd/component.go Outdated Show resolved Hide resolved

surajssd force-pushed the surajssd/add-linkerd2 branch 2 times, most recently from 8f80a0c to e978585 Compare August 24, 2020 14:02

surajssd requested a review from invidian August 25, 2020 11:08

invidian previously approved these changes Aug 25, 2020

View reviewed changes

surajssd dismissed invidian’s stale review via 383cef7 August 26, 2020 06:19

surajssd force-pushed the surajssd/add-linkerd2 branch from e978585 to 383cef7 Compare August 26, 2020 06:19

invidian previously approved these changes Aug 26, 2020

View reviewed changes

surajssd requested a review from knrt10 August 26, 2020 13:50

knrt10 reviewed Aug 26, 2020

View reviewed changes

pkg/components/linkerd/component.go Outdated Show resolved Hide resolved

surajssd added 6 commits August 26, 2020 19:56

assets/components/linkerd2: initial commit

cba9b0b

This commit adds assets for linkerd2 chart at version `stable-2.8.1`. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

ci: Install linkerd component

3f7405e

In AWS install non-HA setup. CI on AWS currently does not have enough resources to run multiple replicas of the linkerd pods. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

e2e test: Add linkerd test

45ebbba

- Add test to verify if the deployments are up and running. - Add test to run `linkerd check` against the apiserver. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

e2e: Add retry logic in calico test

defadfb

If pod creation fails for some reason in first attempt then try again. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

linkerd metrics: Add ServiceMonitor for control plane

4bc0d8b

- Adds ServiceMonitors to monitor linkerd control plane pods. - Adds entry in the components prometheus metrics list to test if the linkerd metrics are being scraped. Signed-off-by: Suraj Deshmukh <suraj@kinvolk.io>

surajssd dismissed invidian’s stale review via 4bc0d8b August 26, 2020 14:27

surajssd force-pushed the surajssd/add-linkerd2 branch from 383cef7 to 4bc0d8b Compare August 26, 2020 14:27

surajssd requested a review from knrt10 August 26, 2020 14:27

knrt10 approved these changes Aug 26, 2020

View reviewed changes

knrt10 requested a review from invidian August 26, 2020 14:30

invidian approved these changes Aug 26, 2020

View reviewed changes

surajssd merged commit df92e35 into master Aug 27, 2020

surajssd deleted the surajssd/add-linkerd2 branch August 27, 2020 06:57

invidian mentioned this pull request Aug 27, 2020

Add linkerd component #697

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

component: Add linkerd #690

component: Add linkerd #690

surajssd commented Jul 1, 2020 •

edited

Loading

invidian commented Jul 3, 2020

surajssd commented Jul 3, 2020

surajssd commented Jul 3, 2020

iaguis commented Jul 3, 2020

surajssd commented Jul 7, 2020

surajssd commented Jul 7, 2020

surajssd commented Jul 7, 2020

invidian left a comment

surajssd commented Aug 25, 2020

invidian left a comment

knrt10 left a comment

surajssd commented Aug 27, 2020

component: Add linkerd #690

component: Add linkerd #690

Conversation

surajssd commented Jul 1, 2020 • edited Loading

invidian commented Jul 3, 2020

surajssd commented Jul 3, 2020

surajssd commented Jul 3, 2020

iaguis commented Jul 3, 2020

surajssd commented Jul 7, 2020

surajssd commented Jul 7, 2020

surajssd commented Jul 7, 2020

invidian left a comment

Choose a reason for hiding this comment

surajssd commented Aug 25, 2020

invidian left a comment

Choose a reason for hiding this comment

knrt10 left a comment

Choose a reason for hiding this comment

surajssd commented Aug 27, 2020

surajssd commented Jul 1, 2020 •

edited

Loading