Retry creating kubeclients in FluentD when error

[samjsong]

Edit: New proposal based on Dominik's comments is to retry for ~4 minutes (with exponential backoff) in the configure step. This should hopefully give enough leeway for transient issue, and throw an exception otherwise, causing the Fluentd worker process to die and restart (after having logged the exception). This will make it way easier to understand the issue than the previous Got exception undefined method.

Based on the issues that @frankreno mentioned some customers were facing with

Got exception undefined method _____ for #<String:0x00007ff19eaf4e60> ______

I found that this is because Ruby does a weird thing where if a method returns an exception, Ruby treats that exception as a String type. In this case, inside connect_kubernetes we call create_client which can return an exception. If for some reason we fail to create this client, we store the exception instead of the client - which means that when we try to call client.getEntity, we get the above undefined method for String error.

By creating the client in the configure step, this means that when the fluentd worker exits and restarts, we never re-initialize the client. By moving connect_kubernetes to start, we will at least try to re-initialize the client.

Unfortunately I spent some time trying to figure out why the initial error happens but haven't been able to figure that out, but hopefully this change at least makes it so that fluentd can fix itself if a transient issue ends.

[sumo-drosiek]

Maybe we could put querying into begin rescue block and reconnect in case of connection is invalid

[samjsong]

I think we could also add new begin rescue blocks where we call kubeclient methods to re-init the clients in that same case

[frankreno]

LGTM - as discussed let's get this lined up for a 1.2.1 release.

[samjsong]

I've made a few changes that I hope will help further

• return nil in create_clients so that @clients[api_version] will return nil rather than a String of the exception thrown
• in the events plugin and enriched metadata plugin service monitor, add a nil check on relevant clients. If nil, attempt to recreate the client before proceeding. There's no guarantee this attempt will succeed (especially if there's a legitimate issue), but as these are infinite loops we'll at least try again soon.
• in fetch_resource, fix the check for client not created. Again, attempt to recreate if nil.

I thought about doing some sort of retry mechanism on client creation, but I wasn't sure how long we'd want to retry - infinitely? If the team thinks adding some sort of retry into create_clients itself, I'd be happy to do that as well

[samjsong]

Hmm, Travis failed even though build passed locally, I’ll check tomorrow

[perk]

Travis vs kubeclient error has been fixed upstream by #870 - please rebase :)

[sumo-drosiek]

🎉

[perk-sumo]

LGTM 👍