Install with the kuberneted operator does not work

[lolovroom]

Hi,

I'm following the documentation, and can't get anything deployed.
I have cloned the project locally and just run the first command :

kustomize build manifests/overlays/generic-rbac | kubectl apply -f -
Error: accumulating resources: 2 errors occurred:
	* accumulateFile error: "accumulating resources from '../../base': '/Users/user/PycharmProjects/kubernetes-crd/manifests/base' must resolve to a file"
	* accumulateDirector error: "recursed accumulation of path '/Users/user/PycharmProjects/kubernetes-crd/manifests/base': accumulating resources: 2 errors occurred:\n\t* accumulateFile error: \"accumulating resources from './common/kustomization.yaml': missing metadata.name in object {map[apiVersion:kustomize.config.k8s.io/v1beta1 kind:Kustomization namespace:crd resources:[./serviceaccount.yaml ./crd.yaml ./configmap.yaml ./deployment.yaml]]}\"\n\t* loader.New error: \"error loading ./common/kustomization.yaml with git: url lacks host: ./common/kustomization.yaml, dir: got file 'kustomization.yaml', but '/Users/user/PycharmProjects/kubernetes-crd/manifests/base/common/kustomization.yaml' must be a directory to be a root, get: invalid source string: ./common/kustomization.yaml\"\n\n"

Anything I missed here ?

After a bit a digging, I pulled the install-in-another-namespace branche, and get something deployed by going into manifest directory, and running :

kustomize build base/common/ | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/chaosexperiments.chaostoolkit.org unchanged
serviceaccount/chaostoolkit-crd created
configmap/chaostoolkit-resources-templates created
deployment.apps/chaostoolkit-crd created

But then the pod is not starting :

kubectl -n crd get pods
NAME                                READY   STATUS   RESTARTS   AGE
chaostoolkit-crd-6b9cf74d69-btzhk   0/1     Error    2          44s

here's the log :

kopf.reactor.running [ERROR ] Root task 'watcher of chaosexperiments.chaostoolkit.org' is failed: 403, message='chaosexperiments.chaostoolkit.org is forbidden: User "system:serviceaccount:crd:chaostoolkit-crd" cannot list resource "chaosexperiments" in API group "chaostoolkit.org" in the namespace "crd"', url=URL('https://X.X.X.X:443/apis/chaostoolkit.org/v1/namespaces/crd/chaosexperiments')

But probably the first issue is the one to look at.

[manukr79]

I am facing the exact same issue. Did u get it working ?

[lolovroom]

it worked when using this fork : r1sharma / kubernetes-crd

[manukr79]

I cloned the forked branch, however when I run basic.yaml example, I get error " serviceaccount:chaostoolkit-crd:chaostoolkit-crd" cannot create resource "namespaces"

[2021-01-18 22:10:55,844] kopf.objects [INFO ] [chaostoolkit-crd/my-chaos-exp] Creation event is processed: 0 succeeded; 1 failed.
[2021-01-18 22:28:49,350] kopf.objects [INFO ] Default PSP for chaostoolkit not found.
[2021-01-18 22:28:49,355] kopf.objects [ERROR ] [chaostoolkit-crd/my-chaos-exp] Handler 'create_chaos_experiment' failed permanently: Failed to create namespace: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '6e30a3c0-2338-4d1a-b817-08c59eb564cf', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 18 Jan 2021 22:28:49 GMT', 'Content-Length': '312'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User "system:serviceaccount:chaostoolkit-crd:chaostoolkit-crd" cannot create resource "namespaces" in API group "" at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}

[2021-01-18 22:28:49,355] kopf.objects [INFO ] [chaostoolkit-crd/my-chaos-exp] Creation event is processed: 0 succeeded; 1 failed.

[CamilleGR]

I got a similar problem using kustomize build manifest/overlays/generic-rbac. With few modification I found that when a parent of namespaces have a namespace tag defined the build fail.

After splitting common folder to create namespaces in a separate kustomization file, I success to build.

This build doesn't work because of the service account chaostoolkit-crd seems to doesn't have access to the chaostoolkit-run namespace.

Maybe the access problem can be solved duplicating the chaostoolkit-crd's serviceaccount (and its roles) in the chaostoolkit-run namespace.

[sathiyajith]

I cloned the forked branch, however when I run basic.yaml example, I get error " serviceaccount:chaostoolkit-crd:chaostoolkit-crd" cannot create resource "namespaces"

[2021-01-18 22:10:55,844] kopf.objects [INFO ] [chaostoolkit-crd/my-chaos-exp] Creation event is processed: 0 succeeded; 1 failed.
[2021-01-18 22:28:49,350] kopf.objects [INFO ] Default PSP for chaostoolkit not found.
[2021-01-18 22:28:49,355] kopf.objects [ERROR ] [chaostoolkit-crd/my-chaos-exp] Handler 'create_chaos_experiment' failed permanently: Failed to create namespace: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '6e30a3c0-2338-4d1a-b817-08c59eb564cf', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 18 Jan 2021 22:28:49 GMT', 'Content-Length': '312'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User "system:serviceaccount:chaostoolkit-crd:chaostoolkit-crd" cannot create resource "namespaces" in API group "" at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}

[2021-01-18 22:28:49,355] kopf.objects [INFO ] [chaostoolkit-crd/my-chaos-exp] Creation event is processed: 0 succeeded; 1 failed.

Im getting the similar error. If anyone solved this error, please mention any solution or work around.

[Lawouach]

I will be looking at this today.

[Lawouach]

One thing that could be useful is to know the version of kubernetes you running. There are times when things evolve in its API that can break with older manifests.

[sathiyajith]

I solved the above error by giving clusterrole and cluster role binding in sec/pod/policy.yaml. But Now Im getting this following error -
[root@cluster-2-master manifests]# kubectl -n chaostoolkit-crd logs chaostoolkit-crd-58c57846cf-l859j
[2021-04-23 07:30:35,717] kopf.reactor.activit [INFO ] Initial authentication has been initiated.
[2021-04-23 07:30:35,810] kopf.activities.auth [INFO ] Activity 'login_via_pykube' succeeded.
[2021-04-23 07:30:35,811] kopf.activities.auth [INFO ] Activity 'login_via_client' succeeded.
[2021-04-23 07:30:35,812] kopf.reactor.activit [INFO ] Initial authentication has finished.
[2021-04-23 07:30:35,924] kopf.engines.peering [WARNING ] Default peering object not found, falling back to the standalone mode.
[2021-04-23 07:30:42,811] kopf.objects [INFO ] Default PSP for chaostoolkit not found.
[2021-04-23 07:30:42,819] kopf.objects [INFO ] Namespace 'chaostoolkit-run' already exists. Let's continue...
[2021-04-23 07:30:42,820] kopf.objects [INFO ] [chaostoolkit-crd/my-chaos-exp] chaostoolkit resources will be created in namespace 'chaostoolkit-run'
[2021-04-23 07:30:42,820] kopf.objects [INFO ] [chaostoolkit-crd/my-chaos-exp] Suffix for resource names will be '-pgfda'
[2021-04-23 07:30:42,916] kopf.objects [INFO ] [chaostoolkit-crd/my-chaos-exp] Created service account
[2021-04-23 07:30:43,117] kopf.objects [INFO ] [chaostoolkit-crd/my-chaos-exp] Created role
[2021-04-23 07:30:43,226] kopf.objects [INFO ] [chaostoolkit-crd/my-chaos-exp] Created rolebinding
[2021-04-23 07:30:43,312] kopf.objects [INFO ] Creating default chaostoolkit-env configmap
[2021-04-23 07:30:43,321] kopf.objects [INFO ] [chaostoolkit-crd/my-chaos-exp] Created experiment's env vars configmap
[2021-04-23 07:30:43,416] kopf.objects [INFO ] Env config map named 'chaostoolkit-env'
[2021-04-23 07:30:43,416] kopf.objects [INFO ] Removing default settings secret volume
[2021-04-23 07:30:43,417] kopf.objects [ERROR ] [chaostoolkit-crd/my-chaos-exp] Handler 'create_chaos_experiment' failed with an exception. Will retry.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/handling.py", line 272, in execute_handler_once
subrefs=subrefs,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/handling.py", line 371, in invoke_handler
**kwargs,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/invocation.py", line 124, in invoke
result = await fn(*args, **kwargs) # type: ignore
File "controller.py", line 101, in create_chaos_experiment
pod_tpl = await create_pod(v1, cm, spec, ns, name_suffix, meta)
File "controller.py", line 397, in run
return await loop.run_in_executor(executor, pfunc)
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "controller.py", line 620, in create_pod
remove_settings_secret(tpl)
File "controller.py", line 198, in remove_settings_secret
for volume in spec["volumes"]:
KeyError: 'volumes'

[sathiyajith]

my kubectl version

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:28:09Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

[Lawouach]

Thanks @sathiyajith !

I'm looking at it this morning with an hopeful fix/release early next week.

[Lawouach]

Hello folks, late but hopefully with useful news. I've fixed a few things and release a new container image of the operator yesterday. Please upgrade to 0.4.0 and let me know if it still fails.

[diegolovison]

Hi, I am using the following

(chaos-experiment) dlovison@dlovison-mac kubernetes-crd % git log --oneline
24a79cd (HEAD -> master, tag: 0.4.0, origin/master, origin/HEAD) Bump version
ae51c0d Bump dependencies
5fbb498 Fix controller losing its resources
a638289 Update container base to Python 3.9
2e9ec5f (tag: 0.3.4) Release v0.3.4

Then, from /Users/dlovison/Documents/GitHub/chaostoolkit-incubator/kubernetes-crd

(chaos-experiment) dlovison@dlovison-mac kubernetes-crd % kustomize build manifests/overlays/generic-rbac | kubectl apply -f -
Error: accumulating resources: accumulation err='accumulating resources from '../../base': '/Users/dlovison/Documents/GitHub/chaostoolkit-incubator/kubernetes-crd/manifests/base' must resolve to a file': recursed accumulation of path '/Users/dlovison/Documents/GitHub/chaostoolkit-incubator/kubernetes-crd/manifests/base': accumulating resources: accumulation err='accumulating resources from './common': '/Users/dlovison/Documents/GitHub/chaostoolkit-incubator/kubernetes-crd/manifests/base/common' must resolve to a file': recursed accumulation of path '/Users/dlovison/Documents/GitHub/chaostoolkit-incubator/kubernetes-crd/manifests/base/common': namespace transformation produces ID conflict: [{"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{"config.kubernetes.io/previousKinds":"Namespace","config.kubernetes.io/previousNames":"chaostoolkit-crd","config.kubernetes.io/previousNamespaces":"_non_namespaceable_"},"name":"chaostoolkit-crd"}}{nsfx:false,beh:unspecified} {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{"config.kubernetes.io/previousKinds":"Namespace","config.kubernetes.io/previousNames":"chaostoolkit-run","config.kubernetes.io/previousNamespaces":"_non_namespaceable_"},"name":"chaostoolkit-crd"}}{nsfx:false,beh:unspecified}]
error: no objects passed to apply

any idea?

[Lawouach]

First guess is a kustomize shenanigan. What version are you using? I think kustomize 4 may have some issues with older formats, would you be okay trying with kuztomize 3 just to check?

[diegolovison]

After the downgrade

(chaos-experiment) dlovison@dlovison-mac kubernetes-crd % kustomize version
{Version:kustomize/v3.10.0 GitCommit:602ad8aa98e2e17f6c9119e027a09757e63c8bec BuildDate:2021-02-10T00:00:50Z GoOs:darwin GoArch:amd64}
(chaos-experiment) dlovison@dlovison-mac kubernetes-crd % kustomize build manifests/overlays/generic-rbac | kubectl apply -f -
Error: accumulating resources: 2 errors occurred:
        * accumulateFile error: "accumulating resources from '../../base': '/Users/dlovison/Documents/GitHub/chaostoolkit-incubator/kubernetes-crd/manifests/base' must resolve to a file"
        * accumulateDirector error: "recursed accumulation of path '/Users/dlovison/Documents/GitHub/chaostoolkit-incubator/kubernetes-crd/manifests/base': accumulating resources: 2 errors occurred:\n\t* accumulateFile error: \"accumulating resources from './common': '/Users/dlovison/Documents/GitHub/chaostoolkit-incubator/kubernetes-crd/manifests/base/common' must resolve to a file\"\n\t* accumulateDirector error: \"recursed accumulation of path '/Users/dlovison/Documents/GitHub/chaostoolkit-incubator/kubernetes-crd/manifests/base/common': namespace transformation produces ID conflict: [{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"Namespace\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"config.kubernetes.io/previousNames\\\":\\\"chaostoolkit-crd\\\",\\\"config.kubernetes.io/previousNamespaces\\\":\\\"_non_namespaceable_\\\"},\\\"name\\\":\\\"chaostoolkit-crd\\\"}}{nsfx:false,beh:unspecified} {\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"Namespace\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"config.kubernetes.io/previousNames\\\":\\\"chaostoolkit-run\\\",\\\"config.kubernetes.io/previousNamespaces\\\":\\\"_non_namespaceable_\\\"},\\\"name\\\":\\\"chaostoolkit-crd\\\"}}{nsfx:false,beh:unspecified}]\"\n\n"


error: no objects passed to apply
(chaos-experiment) dlovison@dlovison-mac kubernetes-crd % 

[Lawouach]

not much better indeed

I'll try again on my side. Thanks for reporting.

[Lawouach]

Hello,

The latest templates should now support Kustomize 4 :)