[kibana] create a secret with es token #1720

jmlrt · 2022-11-04T18:50:39Z

This PR moves the Elasticsearch token for Kibana creation into Helm pre-install hooks and adds it to a K8S secret so it can be used by every pod of the Kibana Deployment.

The K8S secret and all linked resources are cleaned in post-delete hooks.

The call to Elasticsearch and K8S API to create the ES token and convert it to a secret are done in a Node JS script because the only language runtime available in the Kibana Docker image are Node and Bash, but jq command isn't available which exclude Bash script.

The Node JS script is also available in plain JS format to facilitate the review: https://gist.github.com/jmlrt/482b3d984588c1d3e10a84ce5fcd7b3b

Many thanks @pugnascotia for the help on this script 🙏🏻

Follow-up of #1679
Replace #1715
Fixes #1714

jmlrt · 2022-11-08T18:09:07Z

👋🏻 @jbudz @pugnascotia the Node JS script is working fine but could surely be optimized, advises are welcome.

I'm also looking for a better way to handle the case where the Elasticsearch token doesn't exist yet by not having to consider that HTTP 404 is successful (TLDR, when trying to delete an ES token that doesn't exist, it's returning 404) if you have some better idea.

jmlrt · 2022-11-08T20:56:10Z

During the upgrade tests, I think that ES cluster isn't completely ready when kibana is upgraded. Indeed Node JS logs are showing ECONNREFUSED error.

Creating a new Elasticsearch token for Kibana                                                                                     
Cleaning previous token                                                                                                           
DELETE undefined failed:  connect ECONNREFUSED 10.44.11.108:9200                                                                  
Error: connect ECONNREFUSED 10.44.11.108:9200                                                                                     
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16) {                                                           
  errno: -111,                                                                                                                    
  code: 'ECONNREFUSED',                                                                                                           
  syscall: 'connect',                                                                                                             
  address: '10.44.11.108',                                                                                                        
  port: 9200                                                                                                                      
}

However, the K8S job is still in successful state. Need to find a way to fail the JS script in case of this error:

Name:             pre-install-helm-kibana-upgrade-kb-kibana-6dccp                                                                 
Namespace:        default                                                                                                         
Priority:         0                                                                                                               
Service Account:  pre-install-helm-kibana-upgrade-kb-kibana                                                                       
Node:             gke-jmlrt-test-default-pool-5b440e04-0w1m/10.200.0.10                                                           
Start Time:       Tue, 08 Nov 2022 21:51:25 +0100                                                                                 
Labels:           controller-uid=cf0d1165-def4-4377-93b6-fe4164f70475                                                             
                  job-name=pre-install-helm-kibana-upgrade-kb-kibana                                                              
Annotations:      <none>                                                                                                          
Status:           Succeeded                                                                                                       
IP:               10.40.2.20                                       
...

pugnascotia · 2022-11-09T09:30:51Z

kibana/templates/configmap-helm-scripts.yaml

-    source $HOME/.elasticsearch-serviceaccounttoken
+    // Elasticsearch API
+    const esPath = '_security/service/elastic/kibana/credential/token/{{ template "kibana.fullname" . }}';
+    const esUrl = '{{ .Values.elasticsearchHosts }}' + '/' + esPath


Will .Values.elasticsearchHosts always only have a single value?

Good question, it seems it was supposed to support multiple values but a bug that was never fixed prevented it (#603).

As this Kibana chart is supposed to connect to an Elasticsearch cluster deployed via the Elastic Helm charts, all Elasticsearch pods are supposed to be available via a single K8S service that should be used in .Values.elasticsearchHosts.

IMHO, It's not worth fixing #603 and making the script compatible with multiple URLs. I'll update the doc to mention that a single URL (Elasticsearch service URL) is expected for this value, to make it clear.

kibana/templates/configmap-helm-scripts.yaml

pugnascotia · 2022-11-09T09:44:49Z

kibana/templates/configmap-helm-scripts.yaml

+    switch (command) {
+      case 'create':
+        console.log('Creating a new Elasticsearch token for Kibana')
+        createEsToken();


I think it would be cleaner to always return the promises created by requestPromise, so that they are chained together, and put a single top-level .catch(...) here that logs any errors. If that isn't clear, I can put the suggested changes in a rustpad.

Do you mean something like that?

function createEsToken() { // Chaining requests console.log('Cleaning previous token'); return requestPromise(esUrl, esTokenDeleteOptions).then(() => { console.log('Creating new token'); requestPromise(esUrl, esTokenCreateOptions).then(response => { // ... requestPromise(k8sPostSecretUrl, secretCreateOptions, secretData) }); }); } function cleanEsToken() { // Chaining requests console.log('Cleaning token'); return requestPromise(esUrl, esTokenDeleteOptions).then(() => { // Create the k8s secret console.log('Delete K8S secret'); requestPromise(k8sDeleteSecretUrl, secretDeleteOptions) }); } const command = process.argv[2]; switch (command) { case 'create': console.log('Creating a new Elasticsearch token for Kibana') createEsToken().catch(err => { console.error(err); }); break; case 'clean': console.log('Cleaning the Kibana Elasticsearch token') cleanEsToken().catch(err => { console.error(err); }); break; default: console.log('Unknown command'); }

Yes, just make sure you're always returning the value of requestPromise, and the chain of promises will finally resolve in the switch.

done in e3ef0c1

Co-authored-by: Rory Hunter <pugnascotia@users.noreply.github.com>

jmlrt · 2022-11-09T16:15:50Z

I'm also looking for a better way to handle the case where the Elasticsearch token doesn't exist yet by not having to consider that HTTP 404 is successful (TLDR, when trying to delete an ES token that doesn't exist, it's returning 404) if you have some better idea.

improved in 43c08a1

jmlrt · 2022-11-09T16:25:30Z

@pugnascotia PTAL 🙏🏻
I also updated the plain JS version in https://gist.github.com/jmlrt/482b3d984588c1d3e10a84ce5fcd7b3b (and updated the issue description with the good link^^)

jmlrt · 2022-11-09T16:36:13Z

👋🏻 @jbudz @pugnascotia the Node JS script is working fine but could surely be optimized, advises are welcome.

I'm also looking for a better way to handle the case where the Elasticsearch token doesn't exist yet by not having to consider that HTTP 404 is successful (TLDR, when trying to delete an ES token that doesn't exist, it's returning 404) if you have some better idea.

I could reproduce it by setting Elasticsearch replicas to 0 and running the script manually:

kibana@pre-install-kb-kibana-wh4lm:~$ node/bin/node helm-scripts/manage-es-token.js create
Creating a new Elasticsearch token for Kibana
Cleaning previous token
DELETE undefined failed:  connect ECONNREFUSED 10.44.6.135:9200
Error: connect ECONNREFUSED 10.44.6.135:9200
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '10.44.6.135',
  port: 9200
}
kibana@pre-install-kb-kibana-wh4lm:~$ echo $?
0

Despite logging the ECONNREFUSED error, manage-es-token.js still returns success (RC 0).

Need to ensure that the script returns a different RC when it is failing with this error.

jmlrt · 2022-11-09T16:55:00Z

👋🏻 @jbudz @pugnascotia the Node JS script is working fine but could surely be optimized, advises are welcome.
I'm also looking for a better way to handle the case where the Elasticsearch token doesn't exist yet by not having to consider that HTTP 404 is successful (TLDR, when trying to delete an ES token that doesn't exist, it's returning 404) if you have some better idea.

I could reproduce it by setting Elasticsearch replicas to 0 and running the script manually:
kibana@pre-install-kb-kibana-wh4lm:~$ node/bin/node helm-scripts/manage-es-token.js create
Creating a new Elasticsearch token for Kibana
Cleaning previous token
DELETE undefined failed:  connect ECONNREFUSED 10.44.6.135:9200
Error: connect ECONNREFUSED 10.44.6.135:9200
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1187:16) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '10.44.6.135',
  port: 9200
}
kibana@pre-install-kb-kibana-wh4lm:~$ echo $?
0
Despite logging the ECONNREFUSED error, manage-es-token.js still returns success (RC 0).

Need to ensure that the script returns a different RC when it is failing with this error.

should be fixed by 86386f0

kibana/templates/configmap-helm-scripts.yaml

Co-authored-by: Rory Hunter <pugnascotia@users.noreply.github.com>

jmlrt · 2022-11-10T13:11:40Z

ok to test

jmlrt · 2022-11-10T14:33:52Z

https://devops-ci.elastic.co/job/elastic+helm-charts+pull-request+integration-filebeat/1343/FILEBEAT_SUITE=default,KUBERNETES_VERSION=1.24,label=ubuntu/ is a known transient issue, just waiting for a few more review to merge.

framsouza

Great job!! LGTM

mgreau

LGTM 👍

jbudz · 2022-11-15T17:07:38Z

kibana/templates/configmap-helm-scripts.yaml

+          const base64Token = Buffer.from(token, 'utf8').toString('base64');
+
+          // Prepare the k8s secret
+          secretData = JSON.stringify({


Suggested change

secretData = JSON.stringify({

const secretData = JSON.stringify({

whoops I missed the merge, didn't refresh in time. Sorry about the confusion.

(I'll follow up with a PR)

[kibana] create a secret with es token

bacbce1

jmlrt mentioned this pull request Nov 4, 2022

[kibana] initContainer: configure-kibana-token: Back-off restarting failed container: #1714

Closed

fix pre-post-install jobs

2b7524b

jmlrt force-pushed the kibana-pre-install-token branch 2 times, most recently from 9861c77 to 4a590d9 Compare November 8, 2022 17:48

jmlrt marked this pull request as ready for review November 8, 2022 17:51

fix clean command + optimize js script

827ddb3

jmlrt force-pushed the kibana-pre-install-token branch from 4a590d9 to 827ddb3 Compare November 8, 2022 17:53

jmlrt requested review from framsouza, pugnascotia, jbudz and a team November 8, 2022 18:02

jmlrt added bug Something isn't working kibana labels Nov 8, 2022

fix upgrade test

753facd

fix a few bugs

547a349

pugnascotia reviewed Nov 9, 2022

View reviewed changes

kibana/templates/configmap-helm-scripts.yaml Show resolved Hide resolved

pugnascotia reviewed Nov 9, 2022

View reviewed changes

kibana/templates/configmap-helm-scripts.yaml Outdated Show resolved Hide resolved

pugnascotia reviewed Nov 9, 2022

View reviewed changes

kibana/templates/configmap-helm-scripts.yaml Show resolved Hide resolved

pugnascotia reviewed Nov 9, 2022

View reviewed changes

jmlrt and others added 3 commits November 9, 2022 13:01

Update kibana/templates/configmap-helm-scripts.yaml

81ce165

Co-authored-by: Rory Hunter <pugnascotia@users.noreply.github.com>

check if vars exist

ff4a1c6

handle both payload and extra status code

43c08a1

return promises

e3ef0c1

jmlrt requested a review from pugnascotia November 9, 2022 16:24

jmlrt added 3 commits November 9, 2022 17:37

add a note to retrieve the kibana token

5b812e5

exit when catching an error

86386f0

add more retries instead of sleep

cb0d6bb

fixup! add a note to retrieve the kibana token

c08720a

jmlrt mentioned this pull request Nov 9, 2022

Fix kibana's init container configure-kibana-token http request error 409 #1715

Closed

4 tasks

pugnascotia reviewed Nov 10, 2022

View reviewed changes

kibana/templates/configmap-helm-scripts.yaml Outdated Show resolved Hide resolved

pugnascotia reviewed Nov 10, 2022

View reviewed changes

kibana/templates/configmap-helm-scripts.yaml Show resolved Hide resolved

jmlrt and others added 2 commits November 10, 2022 13:15

Update kibana/templates/configmap-helm-scripts.yaml

47ebb41

Co-authored-by: Rory Hunter <pugnascotia@users.noreply.github.com>

Update kibana/templates/configmap-helm-scripts.yaml

84a955d

Co-authored-by: Rory Hunter <pugnascotia@users.noreply.github.com>

pugnascotia approved these changes Nov 10, 2022

View reviewed changes

framsouza approved these changes Nov 15, 2022

View reviewed changes

mgreau approved these changes Nov 15, 2022

View reviewed changes

jmlrt merged commit 90f59ef into elastic:main Nov 15, 2022

jmlrt deleted the kibana-pre-install-token branch November 15, 2022 17:07

jbudz reviewed Nov 15, 2022

View reviewed changes

jmlrt added the v8.5.1 label Nov 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[kibana] create a secret with es token #1720

[kibana] create a secret with es token #1720

jmlrt commented Nov 4, 2022 •

edited

Loading

jmlrt commented Nov 8, 2022

jmlrt commented Nov 8, 2022

pugnascotia Nov 9, 2022

jmlrt Nov 9, 2022

pugnascotia Nov 9, 2022

jmlrt Nov 9, 2022

pugnascotia Nov 9, 2022

jmlrt Nov 9, 2022

jmlrt commented Nov 9, 2022

jmlrt commented Nov 9, 2022

jmlrt commented Nov 9, 2022

jmlrt commented Nov 9, 2022

jmlrt commented Nov 10, 2022

jmlrt commented Nov 10, 2022

framsouza left a comment

mgreau left a comment

jbudz Nov 15, 2022

jbudz Nov 15, 2022

jbudz Nov 15, 2022

jbudz Nov 15, 2022

	secretData = JSON.stringify({
	const secretData = JSON.stringify({

[kibana] create a secret with es token #1720

[kibana] create a secret with es token #1720

Conversation

jmlrt commented Nov 4, 2022 • edited Loading

jmlrt commented Nov 8, 2022

jmlrt commented Nov 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmlrt commented Nov 9, 2022

jmlrt commented Nov 9, 2022

jmlrt commented Nov 9, 2022

jmlrt commented Nov 9, 2022

jmlrt commented Nov 10, 2022

jmlrt commented Nov 10, 2022

framsouza left a comment

Choose a reason for hiding this comment

mgreau left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmlrt commented Nov 4, 2022 •

edited

Loading