Failing to sync cache for ambassador-host

[vsimon]

Attempted to try out the new ambassador-host source based off: #1642

What happened:

After 60 seconds, got a fatal msg="failed to sync cache: timed out waiting for the condition"

What you expected to happen:

The ambassador-host informer syncs successfully.

How to reproduce it (as minimally and precisely as possible):

Added --source=ambassador-host, previously there was just "service" and "ingress" sources that worked correctly (their informer's sync successfully)

Full args:

   containers:
   - args:
     - --log-level=debug
     - --log-format=text
     - --policy=upsert-only
     - --provider=aws
     - --registry=txt
     - --interval=1m
     - --source=service
     - --source=ingress
     - --source=ambassador-host
     - --aws-api-retries=3
     - --aws-batch-change-size=1000

Logs:

 time="2021-02-17T22:47:12Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s ContourLoadBalancerService:heptio-contour/contour SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress ambassador-host] Namespace: AnnotationFilter: LabelFilter
 : FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws GoogleProject: Google
 BatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[] ExcludeDomains:[] ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000
 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: CloudflareProxied:false CloudflareZ
 onesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPa
 ssword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://loca
 lhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:upsert-only Registry:txt TXTOwnerID:default TXTPrefix: TXTSuffix: Interval:1m0s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:debug TXTCa
 cheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2
 136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransI
 PAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false}"
 time="2021-02-17T22:47:12Z" level=info msg="Instantiating new Kubernetes client"
 time="2021-02-17T22:47:12Z" level=debug msg="apiServerURL: "
 time="2021-02-17T22:47:12Z" level=debug msg="kubeConfig: "
 time="2021-02-17T22:47:12Z" level=info msg="Using inCluster-config based on serviceaccount-token"
 time="2021-02-17T22:47:12Z" level=info msg="Created Kubernetes client https://172.20.0.1:443"
 time="2021-02-17T22:47:14Z" level=info msg="Created Dynamic Kubernetes client https://172.20.0.1:443"
 time="2021-02-17T22:48:14Z" level=fatal msg="failed to sync cache: timed out waiting for the condition"
 stream closed

Anything else we need to know?:

Not sure if there are any special settings that are additionally needed if the resources being referenced by the informer are custom resource definitions (Hosts)? Without the --source=ambassador-host argument, external-dns still works.

Environment:

• External-DNS version (use external-dns --version): v0.7.6-73-g0d12207c
• DNS provider: aws
• Others:

[vsimon]

May be related?

❯ kubectl auth can-i get services --as=system:serviceaccount:external-dns:external-dns
yes

❯ kubectl auth can-i get ingresses --as=system:serviceaccount:external-dns:external-dns
yes

❯ kubectl auth can-i get hosts --as=system:serviceaccount:external-dns:external-dns
no

[vsimon]

Was using the latest bitnami external-chart

One of the clusterrole rules for external-dns looked like:

 - apiGroups:
   - extensions
   - networking.k8s.io
   resources:
   - ingresses
   verbs:
   - get
   - list
   - watch

and updated it to:

 - apiGroups:
   - extensions
   - networking.k8s.io
   - getambassador.io
   resources:
   - ingresses
   - hosts
   verbs:
   - get
   - list
   - watch

❯ kubectl auth can-i get hosts --as=system:serviceaccount:external-dns:external-dns
yes

and the external-dns executable is working much better.