ClusterRoleBinding is missing although InstallPlan shows it was created

[Daniel-Fan]

Bug Report

What did you do?
Install ibm-namespace-scope-operator by creating subscription

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  creationTimestamp: '2022-09-28T09:47:44Z'
  generation: 1
  labels:
    operators.coreos.com/ibm-namespace-scope-operator.ibm-common-services: ''
  name: ibm-namespace-scope-operator
  namespace: ibm-common-services
spec:
  channel: v3
  installPlanApproval: Automatic
  name: ibm-namespace-scope-operator
  source: opencloud-operators
  sourceNamespace: openshift-marketplace

What did you expect to see?
The operator is installed successfully and CSV is in succeeded phase

What did you see instead? Under which circumstances?
The operator is in Pending phase and shows error

RequirementsNotMet, one or more requirements couldn't be found

Environment

• operator-lifecycle-manager version:

quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1fcdb11baededc512246014833f6b8950243b09d1da30b85cf62290fe54ceff5

• Kubernetes version information:

• Kubernetes cluster kind: OCP 4.10.25

Possible Solution

This looks like an intermittent issue in OLM. A re-installation should be able to solve the problem.
But we have not apply the workarounds in case further information is required.

Additional context

InstallPlan for operator is completed

apiVersion: operators.coreos.com/v1alpha1
kind: InstallPlan
metadata:
  generateName: install-
  resourceVersion: '79059'
  name: install-s4dhz
  uid: 39f98c71-3239-46f6-9dae-0810e5244c8d
  creationTimestamp: '2022-09-28T09:47:44Z'
  generation: 1
  managedFields:
    - apiVersion: operators.coreos.com/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:generateName': {}
          'f:ownerReferences':
            .: {}
            'k:{"uid":"6c07a62f-82f8-48a8-b704-d99ebdfc2caf"}': {}
        'f:spec':
          .: {}
          'f:approval': {}
          'f:approved': {}
          'f:clusterServiceVersionNames': {}
          'f:generation': {}
      manager: catalog
      operation: Update
      time: '2022-09-28T09:47:44Z'
    - apiVersion: operators.coreos.com/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:labels':
            .: {}
            'f:operators.coreos.com/ibm-namespace-scope-operator.ibm-common-services': {}
      manager: olm
      operation: Update
      time: '2022-09-28T09:47:45Z'
    - apiVersion: operators.coreos.com/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          .: {}
          'f:bundleLookups': {}
          'f:catalogSources': {}
          'f:conditions': {}
          'f:phase': {}
          'f:plan': {}
          'f:startTime': {}
      manager: catalog
      operation: Update
      subresource: status
      time: '2022-09-28T09:47:57Z'
  namespace: ibm-common-services
  ownerReferences:
    - apiVersion: operators.coreos.com/v1alpha1
      blockOwnerDeletion: false
      controller: false
      kind: Subscription
      name: ibm-namespace-scope-operator
      uid: 6c07a62f-82f8-48a8-b704-d99ebdfc2caf
  labels:
    operators.coreos.com/ibm-namespace-scope-operator.ibm-common-services: ''
spec:
  approval: Automatic
  approved: true
  clusterServiceVersionNames:
    - ibm-namespace-scope-operator.v1.13.3
  generation: 1
status:
  bundleLookups:
    - catalogSourceRef:
        name: opencloud-operators
        namespace: openshift-marketplace
      identifier: ibm-namespace-scope-operator.v1.13.3
      path: >-
        icr.io/cpopen/ibm-namespace-scope-operator-bundle@sha256:b9f34ad34758d618761b9c7b3908af50544407b56867f60f7616a6abf621acbb
      properties: >-
        {"properties":[{"type":"olm.gvk","value":{"group":"operator.ibm.com","kind":"NamespaceScope","version":"v1"}},{"type":"olm.package","value":{"packageName":"ibm-namespace-scope-operator","version":"1.13.3"}}]}
      replaces: ibm-namespace-scope-operator.v1.13.2
  catalogSources: []
  conditions:
    - lastTransitionTime: '2022-09-28T09:47:57Z'
      lastUpdateTime: '2022-09-28T09:47:57Z'
      status: 'True'
      type: Installed
  phase: Complete
  plan:
    - resolving: ibm-namespace-scope-operator.v1.13.3
      resource:
        group: operators.coreos.com
        kind: ClusterServiceVersion
        manifest: >-
          {"kind":"ConfigMap","name":"99f33457374732ade8da76ad7ce904777de2f1379c46b242104015b5c626302","namespace":"openshift-marketplace","catalogSourceName":"opencloud-operators","catalogSourceNamespace":"openshift-marketplace","replaces":"ibm-namespace-scope-operator.v1.13.2","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"NamespaceScope\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"ibm-namespace-scope-operator\",\"version\":\"1.13.3\"}}]}"}
        name: ibm-namespace-scope-operator.v1.13.3
        sourceName: opencloud-operators
        sourceNamespace: openshift-marketplace
        version: v1alpha1
      status: Created
    - resolving: ibm-namespace-scope-operator.v1.13.3
      resource:
        group: apiextensions.k8s.io
        kind: CustomResourceDefinition
        manifest: >-
          {"kind":"ConfigMap","name":"99f33457374732ade8da76ad7ce904777de2f1379c46b242104015b5c626302","namespace":"openshift-marketplace","catalogSourceName":"opencloud-operators","catalogSourceNamespace":"openshift-marketplace","replaces":"ibm-namespace-scope-operator.v1.13.2","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"NamespaceScope\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"ibm-namespace-scope-operator\",\"version\":\"1.13.3\"}}]}"}
        name: namespacescopes.operator.ibm.com
        sourceName: opencloud-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Created
    - resolving: ibm-namespace-scope-operator.v1.13.3
      resource:
        group: ''
        kind: ServiceAccount
        manifest: >-
          {"kind":"ConfigMap","name":"99f33457374732ade8da76ad7ce904777de2f1379c46b242104015b5c626302","namespace":"openshift-marketplace","catalogSourceName":"opencloud-operators","catalogSourceNamespace":"openshift-marketplace","replaces":"ibm-namespace-scope-operator.v1.13.2","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"NamespaceScope\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"ibm-namespace-scope-operator\",\"version\":\"1.13.3\"}}]}"}
        name: ibm-namespace-scope-operator
        sourceName: opencloud-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Created
    - resolving: ibm-namespace-scope-operator.v1.13.3
      resource:
        group: rbac.authorization.k8s.io
        kind: ClusterRole
        manifest: >-
          {"kind":"ConfigMap","name":"99f33457374732ade8da76ad7ce904777de2f1379c46b242104015b5c626302","namespace":"openshift-marketplace","catalogSourceName":"opencloud-operators","catalogSourceNamespace":"openshift-marketplace","replaces":"ibm-namespace-scope-operator.v1.13.2","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"NamespaceScope\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"ibm-namespace-scope-operator\",\"version\":\"1.13.3\"}}]}"}
        name: ibm-namespace-scope-operator.v1.13.3-5dc695597b
        sourceName: opencloud-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Created
    - resolving: ibm-namespace-scope-operator.v1.13.3
      resource:
        group: rbac.authorization.k8s.io
        kind: ClusterRoleBinding
        manifest: >-
          {"kind":"ConfigMap","name":"99f33457374732ade8da76ad7ce904777de2f1379c46b242104015b5c626302","namespace":"openshift-marketplace","catalogSourceName":"opencloud-operators","catalogSourceNamespace":"openshift-marketplace","replaces":"ibm-namespace-scope-operator.v1.13.2","properties":"{\"properties\":[{\"type\":\"olm.gvk\",\"value\":{\"group\":\"operator.ibm.com\",\"kind\":\"NamespaceScope\",\"version\":\"v1\"}},{\"type\":\"olm.package\",\"value\":{\"packageName\":\"ibm-namespace-scope-operator\",\"version\":\"1.13.3\"}}]}"}
        name: ibm-namespace-scope-operator.v1.13.3-5dc695597b
        sourceName: opencloud-operators
        sourceNamespace: openshift-marketplace
        version: v1
      status: Created
  startTime: '2022-09-28T09:47:56Z'

ClusterRoleBinding shows created in InstallPlan

ClusterRoleBinding is missing in cluster

OLM logs

time="2022-09-28T19:03:36Z" level=info msg="requirements were not met" csv=ibm-namespace-scope-operator.v1.13.3 id=pyh1/ namespace=ibm-common-services phase=Pending
E0928 19:03:36.495694 1 queueinformer_operator.go:290] sync {"update" "ibm-common-services/ibm-namespace-scope-operator.v1.13.3"} failed: requirements were not met
time="2022-09-28T19:03:36Z" level=info msg="requirements were not met" csv=ibm-namespace-scope-operator.v1.13.3 id=ux/Ji namespace=ibm-common-services phase=Pending
E0928 19:03:36.738711 1 queueinformer_operator.go:290] sync {"update" "ibm-common-services/ibm-namespace-scope-operator.v1.13.3"} failed: requirements were not met
{"level":"error","ts":1664391699.5386794,"logger":"controllers.operatorcondition","msg":"Error ensuring OperatorCondition Deployment EnvVars","request":"ibm-common-services/ibm-namespace-scope-operator.v1.13.3","error":"Deployment.apps \"ibm-namespace-scope-operator\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1664391699.5388203,"logger":"controller.operatorcondition","msg":"Reconciler error","reconciler group":"operators.coreos.com","reconciler kind":"OperatorCondition","name":"ibm-namespace-scope-operator.v1.13.3","namespace":"ibm-common-services","error":"Deployment.apps \"ibm-namespace-scope-operator\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

CSV for namespace-scope oeprator

apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  annotations:
    olm.skipRange: <1.13.3
    operators.operatorframework.io/builder: operator-sdk-v1.19.0+git
    operators.operatorframework.io/project_layout: go.kubebuilder.io/v2
    olm.targetNamespaces: ibm-common-services
    operatorframework.io/properties: >-
      {"properties":[{"type":"olm.gvk","value":{"group":"operator.ibm.com","kind":"NamespaceScope","version":"v1"}},{"type":"olm.package","value":{"packageName":"ibm-namespace-scope-operator","version":"1.13.3"}}]}
    repository: 'https://github.com/IBM/ibm-namespace-scope-operator'
    support: IBM
    alm-examples: |-
      [
        {
          "apiVersion": "operator.ibm.com/v1",
          "kind": "NamespaceScope",
          "metadata": {
            "name": "namespacescope"
          },
          "spec": {
            "namespaceMembers": [
              "ibm-common-services",
              "default"
            ],
            "restartLabels": {
              "intent": "projected"
            }
          }
        }
      ]
    capabilities: Seamless Upgrades
    olm.operatorNamespace: ibm-common-services
    containerImage: >-
      icr.io/cpopen/ibm-namespace-scope-operator@sha256:46eedea2d5f34a609015ceaa1dae24feee4f5bd334382917dc1bcfcf96cd00cc
    createdAt: '2022-08-12T13:52:44Z'
    olm.operatorGroup: ibm-common-services-operators
  resourceVersion: '79098'
  name: ibm-namespace-scope-operator.v1.13.3
  uid: 9c65811d-2005-40d5-b819-4453615c4594
  creationTimestamp: '2022-09-28T09:47:56Z'
  generation: 1
  managedFields:
    - apiVersion: operators.coreos.com/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:createdAt': {}
            'f:alm-examples': {}
            'f:capabilities': {}
            'f:olm.skipRange': {}
            .: {}
            'f:containerImage': {}
            'f:operators.operatorframework.io/project_layout': {}
            'f:operatorframework.io/properties': {}
            'f:operators.operatorframework.io/builder': {}
            'f:support': {}
            'f:repository': {}
        'f:spec':
          'f:version': {}
          'f:maturity': {}
          'f:provider':
            .: {}
            'f:name': {}
          'f:links': {}
          'f:install':
            .: {}
            'f:spec':
              .: {}
              'f:clusterPermissions': {}
              'f:deployments': {}
            'f:strategy': {}
          'f:maintainers': {}
          'f:description': {}
          'f:installModes': {}
          'f:minKubeVersion': {}
          'f:icon': {}
          'f:customresourcedefinitions':
            .: {}
            'f:owned': {}
          .: {}
          'f:relatedImages': {}
          'f:cleanup':
            .: {}
            'f:enabled': {}
          'f:apiservicedefinitions': {}
          'f:replaces': {}
          'f:displayName': {}
          'f:keywords': {}
      manager: catalog
      operation: Update
      time: '2022-09-28T09:47:56Z'
    - apiVersion: operators.coreos.com/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:olm.operatorGroup': {}
            'f:olm.operatorNamespace': {}
            'f:olm.targetNamespaces': {}
          'f:labels':
            .: {}
            'f:operators.coreos.com/ibm-namespace-scope-operator.ibm-common-services': {}
      manager: olm
      operation: Update
      time: '2022-09-28T09:47:57Z'
    - apiVersion: operators.coreos.com/v1alpha1
      fieldsType: FieldsV1
      fieldsV1:
        'f:status':
          'f:lastUpdateTime': {}
          'f:requirementStatus': {}
          'f:phase': {}
          'f:lastTransitionTime': {}
          'f:conditions': {}
          .: {}
          'f:cleanup': {}
          'f:message': {}
          'f:reason': {}
      manager: olm
      operation: Update
      subresource: status
      time: '2022-09-28T09:47:58Z'
  namespace: ibm-common-services
  labels:
    operators.coreos.com/ibm-namespace-scope-operator.ibm-common-services: ''
spec:
  customresourcedefinitions:
    owned:
      - description: NamespaceScope is the Schema for the namespacescopes API
        displayName: Namespace Scope
        kind: NamespaceScope
        name: namespacescopes.operator.ibm.com
        version: v1
  relatedImages:
    - image: >-
        icr.io/cpopen/ibm-namespace-scope-operator@sha256:46eedea2d5f34a609015ceaa1dae24feee4f5bd334382917dc1bcfcf96cd00cc
      name: IBM_NAMESPACE_SCOPE_OPERATOR_IMAGE
  cleanup:
    enabled: false
  apiservicedefinitions: {}
  keywords:
    - IBM
    - Cloud
  displayName: IBM NamespaceScope Operator
  provider:
    name: IBM
  maturity: alpha
  installModes:
    - supported: true
      type: OwnNamespace
    - supported: true
      type: SingleNamespace
    - supported: false
      type: MultiNamespace
    - supported: true
      type: AllNamespaces
  version: 1.13.3
  icon:
    - base64data: >-
        iVBORw0KGgoAAAANSUhEUgAAAK8AAACvCAMAAAC8TH5HAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAB1UExURQAAAJGS77CC4pCS75yM64uV8pSQ7puA85OV87OB4auF5Hyd+H2c936b9n6b94Ca9n+b9n+b9n+b9qOJ56SI55yM6qSI536b96aH5q2D45mN64OZ9ZWQ7oyU8XWg+6uG5oqg/p6L6m+k/ZuY+3mr/6qQ9LqM80D8C0oAAAAbdFJOUwA67R4KKxMBBP6ak6vZgVtJxG5ot+hQ7YDVkwC2C58AAAuSSURBVHja7ZyJerK8EoCDCSTKjoiIS13of/+XeGYm4NLKrvj1OYxt7aa8TiazJZGxSSaZZJJJJvmcSCn/Eq7Cz79DLJk0rb+kXdM9nz0m/4p2mZufz3lAZvEn1HsGye2J9128h7/Gezj8Nd7D3+I9/xu8SjWHrS76bfN8A+NsYxjowCvbPN+QSGB6kWi6QHteyQLPfx+wYsH2eHSthgu05lXMy/PceRcwxtnjdnts4mjLq5hBceVdcVsya71FMeov0JIXMuQwR+DoXX5EMgf0uz2GrDYbb8mrmE+4Z/NdvDCApN+jX3uFdrySqfW70wzFbFLwWtVNkXa8ONlIvfx9Dk0xSyvYq0NpxasYJ9o8emcUVCw6EjGvuUpLXgfVm9cP1fAZp1yyCKeGBf8pB96g9jUZ57c6s1vIIAUfjXqY9eFg1yiuKJnOECzeW+TJm0+rxRGGWfcP7/dld8bZwqcp/dJqIs9hrJIJ/JD2abV5j1StfJn1/pofo/Kx0ae1KfAO7/Vld7anfVpf28M5kKPDc9kYLRW4RDhIwYV/PozVUAF39Qre3BmrvsM04nisjHHyJlUjZEOefuBj8UIA81zHfGJ84BYeHAP9LKseP1r5LNnvOlHeXJgqRZbUPzT97PHvBVb48VCX09W54du2u3ZJwjD0It/gqmCue/yoolm4b7tQjmohh7cGAWzHC8x/qOFOZmBG4bbERDkQrVYyiGP7iPwPLGrgsAofYbePonEJ2CHxAuvjxEjLvfUj7J1BaP0irY3i888SA63l3alWgwKjbXueZztOSBoucOE33huIZdsWHChXRds72O069PyHhSEBDiOynbAEBiGreCGJKoa5zT8GVBzt4QNgXc+wbq4YvW+hSMkDYNa4EYihWqlYtmouSsYTo4XvgWezHKDcI+7xuPbMMp7JH0GEfhZGRMDIG5FRtLG1IGCNvTp/d9nFZhMx/DXYH/cgSBv6SscM+Tyf0P450Lw+iCmbOGAMonOeO/XlMyTjgAsfmWAN9Y53RFy0hDAovXBDSBFBVAIHDdUJ2lre3J6AVG9Hcln5NQyKCUcrd390g5/BtjpNR2KNGwTVpRDSmk6et6jwCv0ScVhpxopxl3DBIjzVjrYk5gVuEPAaw7UP+aFV+0ex5Aq8y/hTYhiE/UXjhibrlBUisUm8hmHwqujuH3IqQLA/0dT+Af8Q34hT8du3QXlR4nrdkxhJ0554nwAXhpvj+hLUo2u/zWoJM1aXy70ZP8e97APWJ+WGbN1AXNP8tedAasM96PLu4Ik2jhpHZLkqgdGM5TNjuKzNnhkiUmneH8CSCe9wpXV429HDlCu7GcV9JwemWoEbWr3rGZx2iMs5F4+T3S1p89DoYGvkUeLCKC67m+uBsVwVuGpI+QVohGtZ6rHrU+Cu/UaP/ps4KY3iWhlipwNwd4Arh1WLCIy4lpA/2yiF4XZ9ehgMuaRgt7r6FMWiC9DuL64YWtyCrQKuEOLe1iJsG+eO2W8eo+POdrvVtdULrgG0Dbg76xW1uCDcm5GCguzDAeNlz0qPqgfzGunJeAl4aOug6KYQ7l2WhI7DZEMqZ7L5a1uBZWTQF3/QVHvmUosOBX0ZVkbfkgNtDYCbDcDVsIKbQYCJBCY/gak7FHQh+bqiX7LwsnuYfr1gqUTCUsPWgsWdF1H2I1/ZoYBMSLs3o3/blyke+FRiEPE9c1Huq9dpV60GWQNmvybSIrCnee0SGIlDJzJfVzwrttTq7bfkUNCSzV71a19pScNOGHrmi9pWV/Uue6lXYpEcBFfgslSOPG0MBTASc/YK3455PEqvyYY5r0G4AeH6gWHqSCyVxQ2s9ksJw9B/ATBYVUy8fdRL6ZhhlPo1HpIyHelM38OmCuA6oWvzwTah69DTbiW6qxdMCdPdAIGLbrC8lyIimxHRgrhQcA+cdoqluxXc0u7qhcTGNBAYeKkB9CTASfJjVuTo7mvoRsO676Ci+LRanVbd91YgLggp2GI1/kpRq7MAXnuDjBhC8Qpkl3UepwIXgblseDQq2XBcUK8bru0hGgbni7ynzrMNs1xOuJDmNQMAsfAI2B0CjOaAvKuuK2aES8C8XU8Sn98H9SKw12/SwfwVzNyArOLOL1lxEpO37/lKFujlpW3UfTSZwpxaQCkXb+JVd3OAAg1xrQ4vFGzC0MDrbuvLSGtRiSVYuonjeNU5MxMWAVudZzct1azdLmUXzGZLV7BCySxG6Zrq4MsFXqv79A7WiLu1OwwLFgElr7VA3LQjLtZnCCx7+KNo7a4BuG3lhRmKWXQ0LME40Gbxsqt6BQH3arExZ+viCl67Ib1rGHFLQPIQL7JFnHTjRfUCb68whR1mXM3dttpjcWvIAS6uNCRxlmVxxypeCVJw3wjl0/LzmrfaVG4kBgFT6ge57wJ4M7OTfmlNS4j+McpB4G2rTfBGkhAwp2UcWfB2cw/FFogBKQvxrhtTLMnMZYJiFG4eeLM0zVLRg3dIzmJvAbfRgiXjS81rXfeBLIE3TTuVQneZeH8Fb4HXFQ0rcGKJcsNFXsRdduYdViSQBQNy0LCilaSIu+R3TeqP8KKLQAXXzjgw3hR5l3erFvoldOOVr9Cv5eK6v1tzXch0UZfLNGEPvGQi3fU7tMi1m45PgCtb4Nin974Lftmd9yUtJZ94q/NgUG9KvA9rWOjgwKATMTqv3mpcbcDgQxaLRbpYyp+89/5tLMF98GTAVZsP4LfpAuXRYnALBwof+0AxejR0EVVpO4ARbvpz96D1GV7FvNoJB4lNDLiQOKofIQSTicQcnzeq5ZUsxTpi8ctQJeVrJmNj8wbEWxHhYNxjXff8UiT1vww1Oq9R59Dgz1gGb5Kff5a62jA/4tD222Ml75J4zd+8uglmfcQB76s2nktsM2w2z8p2yamWG90eTNrd9ly/ALnAtlP8LO5a1FdSo9sv7h3cVvGqGHkXT9Sr+3ZcjO4faNNYUMErkHf2tIeuqBNhjc0bHXEDoVHBa20qeRm1liw1Mq9H29z68Ard+hs7f0BzWD/3S8g7q+TV3RohR8VVLqq34pgR2G8NL9O8alx3Rrvy7Cr3q2LkXTyPClrBY55JgPqCthFGVbxsgbxxRd2jxKCGTS/zpelW0beD8pB4NxVhVw7t2HSvj0m9lfUx5A/zzWw2q0yPHzYHjWEOuDXvWLnhAtL1Gah3XrWsImkL/WjAkoX7au+r00bQ7my+qFr4ekETpFvyUGsOKOAgZrNNZaE2InCx9XF/qVmFQwNGBVevs42n31K9+5oqFxw0GURc22UayXjBenHrY1Z7UJ/FpOCkRsFjWe+SNsLuef2xCm0QMfvwe60pxnGf5v7iNTR/xWZWb8GjWcOFgBtK3FLBM+uTCpatd5aigue1Pngs4yVcp8VphmT+YYuQGIhxm/Fu37w+j0mPBk4+BIy4ett8q52lGJTneJsbHwHGwx/FQYp2Q6wtogCWH8DNLtdt0S1Pi6RICx8JG1nFCluOV9yWLgrrjAI4HfVQNtYu5emw9ri0EyZGWpCNORYxvVuAGZeHgLIuEVZB5UnAqGLryfsLvDx31Gfa6czSSW+D7XRFVZgEyizlRfEm3yJFSaiM+HQ5Ee5ll3SNVgCczkvi+SJ5c+PMMtIV0BLu6RL32P8Lry8pcVHJcZoYlniDcCNJ49Xp+/uk5QK20PP0kLWYP8qsg2zuvl/VyAlQS1bQ7SnjfQ814O7WeF4jX/P/5l//fT2V77svePeNd/gFNam/FN/eZPd9io0B/ojOwMWVsA8/wO1RZvc/nOgTbqfi7okAfDbUe+KDjcVsPq9X81eJPK/g/So476kfWUG1S6vjmcIqYpGkGwT7r4t8FfffdIP7ajmdNlnC2Qto2fWNtixjudRr4a+VLF0uTa4vJF8XKuXbg/Hr33TjffKn3gp/kkkmmWSSSSaZZJJJJplkkkkmmWSS/yf5H6HANgUotAMHAAAAAElFTkSuQmCC
      mediatype: image/png
  minKubeVersion: 1.19.0
  links:
    - name: IBM Namespace Scope Operator
      url: 'https://github.com/IBM/ibm-namespace-scope-operator'
  install:
    spec:
      clusterPermissions:
        - rules:
            - apiGroups:
                - '*'
              resources:
                - '*'
              verbs:
                - create
                - delete
                - get
                - list
                - patch
                - update
                - watch
                - deletecollection
            - apiGroups:
                - rbac.authorization.k8s.io
              resources:
                - roles
              verbs:
                - escalate
                - bind
          serviceAccountName: ibm-namespace-scope-operator
      deployments:
        - label:
            app.kubernetes.io/instance: ibm-namespace-scope-operator
            app.kubernetes.io/managed-by: ibm-namespace-scope-operator
            app.kubernetes.io/name: ibm-namespace-scope-operator
          name: ibm-namespace-scope-operator
          spec:
            replicas: 1
            selector:
              matchLabels:
                name: ibm-namespace-scope-operator
            strategy: {}
            template:
              metadata:
                annotations:
                  productID: 068a62892a1e4db39641342e592daa25
                  productMetric: FREE
                  productName: IBM Cloud Platform Common Services
                creationTimestamp: null
                labels:
                  app.kubernetes.io/instance: ibm-namespace-scope-operator
                  app.kubernetes.io/managed-by: ibm-namespace-scope-operator
                  app.kubernetes.io/name: ibm-namespace-scope-operator
                  name: ibm-namespace-scope-operator
              spec:
                affinity:
                  nodeAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                      nodeSelectorTerms:
                        - matchExpressions:
                            - key: kubernetes.io/arch
                              operator: In
                              values:
                                - amd64
                                - ppc64le
                                - s390x
                containers:
                  - command:
                      - /namespace-scope-operator-manager
                    env:
                      - name: OPERATOR_NAME
                        value: ibm-namespace-scope-operator
                      - name: OPERATOR_NAMESPACE
                        valueFrom:
                          fieldRef:
                            apiVersion: v1
                            fieldPath: metadata.namespace
                    image: >-
                      icr.io/cpopen/ibm-namespace-scope-operator@sha256:46eedea2d5f34a609015ceaa1dae24feee4f5bd334382917dc1bcfcf96cd00cc
                    imagePullPolicy: Always
                    name: ibm-namespace-scope-operator
                    resources:
                      limits:
                        cpu: 500m
                        memory: 512Mi
                      requests:
                        cpu: 100m
                        memory: 200Mi
                    securityContext:
                      allowPrivilegeEscalation: false
                      capabilities:
                        drop:
                          - ALL
                      privileged: false
                      readOnlyRootFilesystem: true
                      runAsNonRoot: true
                serviceAccountName: ibm-namespace-scope-operator
                terminationGracePeriodSeconds: 10
    strategy: deployment
  maintainers:
    - email: support@ibm.com
      name: IBM Support
  description: >-
    This operator automates the extension of operator watch and service account
    permission scope to other namespaces in an openshift cluster.
  replaces: ibm-namespace-scope-operator.v1.13.2
status:
  cleanup: {}
  conditions:
    - lastTransitionTime: '2022-09-28T09:47:58Z'
      lastUpdateTime: '2022-09-28T09:47:58Z'
      message: requirements not yet checked
      phase: Pending
      reason: RequirementsUnknown
    - lastTransitionTime: '2022-09-28T09:47:58Z'
      lastUpdateTime: '2022-09-28T09:47:58Z'
      message: one or more requirements couldn't be found
      phase: Pending
      reason: RequirementsNotMet
  lastTransitionTime: '2022-09-28T09:47:58Z'
  lastUpdateTime: '2022-09-28T09:47:58Z'
  message: one or more requirements couldn't be found
  phase: Pending
  reason: RequirementsNotMet
  requirementStatus:
    - group: operators.coreos.com
      kind: ClusterServiceVersion
      message: CSV minKubeVersion (1.19.0) less than server version (v1.23.5+012e945)
      name: ibm-namespace-scope-operator.v1.13.3
      status: Present
      version: v1alpha1
    - group: apiextensions.k8s.io
      kind: CustomResourceDefinition
      message: CRD is present and Established condition is true
      name: namespacescopes.operator.ibm.com
      status: Present
      uuid: 05b57f41-2fa7-4411-9de5-e037ee248b8b
      version: v1
    - dependents:
        - group: rbac.authorization.k8s.io
          kind: PolicyRule
          message: >-
            cluster
            rule:{"verbs":["create","delete","get","list","patch","update","watch","deletecollection"],"apiGroups":["*"],"resources":["*"]}
          status: NotSatisfied
          version: v1
        - group: rbac.authorization.k8s.io
          kind: PolicyRule
          message: >-
            cluster
            rule:{"verbs":["escalate","bind"],"apiGroups":["rbac.authorization.k8s.io"],"resources":["roles"]}
          status: NotSatisfied
          version: v1
      group: ''
      kind: ServiceAccount
      message: Policy rule not satisfied for service account
      name: ibm-namespace-scope-operator
      status: PresentNotSatisfied
      version: v1

[bluzarraga]

I am seeing this bug relatively frequently on a couple different clusters. Is there any plan to resolve this or at least make the bug more clear? The install plan reports that the clusterrole and clusterrolebinding are present when they do not exist on the cluster. Is it possible to include some kind of check in OLM to verify the presence of these resources and then re-attempt the install/creation of these resources?

[joelanford]

The install plan was designed as a one-shot style API. It sounds like the install plan either found or successfully created those objects at the time it ran, but OLMv0 will not re-create them if they were deleted after the fact.

We know this is a limitation of OLMv0 and have accounted for it in the OLMv1 design that we are currently implementing. Between the architecture of the API and our attention turned primarily toward OLMv1, there is no plan to make changes in this area of OLMv0.

[joelanford]

However, if the bug is "I can prove that the InstallPlan reconciliation lied about creating the objects in the first place", that is something we would consider fixing.

[bluzarraga]

@joelanford I am not sure if the resource is created and later deleted or if it is never present and the install plan is "lying" but I will try to find evidence to prove. The difficult part is that it is intermittent and only recognizable after whatever is relying on these clusterrole/bindings is far enough along in its deployment process to start failing due to lack of permissions outside of their home namespace

[bluzarraga]

@joelanford We have been able to reproduce this issue consistently on the same cluster (and occasionally on different clusters) over the last week. To verify whether or not the ClusterRole/ClusterRoleBindings were created, we checked the kube-apiserver and openshift-apiserver audit logs. Searching for the ClusterRole/ClusterRoleBinding (which share a name) in these audit logs returns 0 results. It is not mentioned as a create or delete event which we would expect to see if it was successfully created by the install plan and somehow deleted by something else. To verify this, we created/edited/deleted a dummy ClusterRole/ClusterRoleBinding and saw it included as a create/edit/delete event in the audit logs.

Open questions:

• Can we check why the InstallPlan didn’t create the ClusterRole/ClusterRoleBinding?
• Is there a suggested workaround from OLM side of things?