does-argo-events-support-azure-hub-as-eventsource ?

[sky29]

does argo-events supports azure-hub as eventsource ?

I can see one sample code here:

https://github.com/argoproj/argo-events/blob/master/examples/event-sources/azure-events-hub.yaml

but it is not working for me. I tried with below parameters:

FQDN : "namespace".servicebus.windows.net
sharedAccessKeyName: Shared access policies --> "Policy"
sharedAccessKey: "Primary Key"

I also checked eventsources - setup page, azure-hub instructions are not mentioned here:
https://argoproj.github.io/argo-events/eventsources/setup/webhook/

I am able to get connection string successfully using azure cli, as mentioned in below link:
https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-get-connection-string#getting-the-connection-string-with-azure-cli

My argo EventSource pod is in "CrashLoopBackOff" stage & log shows something like this:

started processing the Azure Events Hub
retrieving the shared access key name
retrieving the shared access key...
connecting to the hub...
gathering the hub runtime information...
failed to start listening eventsource
NATS connection lost
failed to start eventsource server

[sky29]

I have upgraded argo-events to latest version - v1.3.0 today & trying to integrate azure-event-hub.

but my eventsource pod is now crashing with different error:
"error":"open /tmp/graft.log: no such file or directory"

Logs:

{"level":"info","ts":1619095607.6588032,"logger":"argo-events.eventsource","caller":"cmd/main.go:63","msg":"starting eventsource server","eventSourceName":"azure-events-hub","version":"vv1.3.0+7591146.dirty"}
{"level":"info","ts":1619095607.6589344,"logger":"argo-events.eventsource","caller":"metrics/metrics.go:172","msg":"starting metrics server","eventSourceName":"azure-events-hub"}
{"level":"fatal","ts":1619095607.6648698,"logger":"argo-events.eventsource","caller":"leaderelection/leaderelection.go:111","msg":"failed to new a node","eventSourceName":"azure-events-hub","error":"open /tmp/graft.log: no such file or directory","stacktrace":"github.com/argoproj/argo-events/common/leaderelection.(*natsEventBusElector).RunOrDie\n\t/home/runner/work/argo-events/argo-events/common/leaderelection/leaderelection.go:111\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:293\nmain.main\n\t/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:65\nruntime.main\n\t/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/proc.go:203"}

basically it is breaking at this stage:
https://github.com/argoproj/argo-events/blob/master/common/leaderelection/leaderelection.go

any idea, what is causing this error ?

[whynowy]

Could you share the detail how you install v1.3.0?

The error says it could not find /tmp dir. EventSource Pod should have a emptyDir volume mounted to /tmp automatically, that is conducted by eventsource-controller. Could you verify if the event source pod has that spec? Also your eventsource-controller is also running v1.3.0, right?

[sky29]

@whynowy

I have fixed it with some manual steps. I have mentioned the details here:
#874 (comment)

[Your comment] EventSource Pod should have a emptyDir volume mounted to /tmp automatically, that is conducted by eventsource-controller ---> No, this is not working for me (I am not sure about the reason, even I have controllers running with v1.3.0), so I manually edited/patched kubernetes deployment file (of EventSource) & added volume + volume-mount at /tmp

[whynowy]

Are you able to firmly reproduce it with whatever event source (let's say a simple calendar one like below)? Also, could you check if there's anything abnormal in eventsource-controller log?

kind: EventSource
metadata:
  name: calendar
spec:
  calendar:
    example1:
      interval: 30s

[sky29]

@whynowy I can reproduce the same error with calendar event source also:

My EventSource File:

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: calander
  namespace: argo-events
spec:
  eventBusName: default
  calendar:
    example1:
      interval: 30s

Logs:

kubectl logs -n argo-events

{"level":"info","ts":1619110843.5463223,"logger":"argo-events.eventsource","caller":"cmd/main.go:63","msg":"starting eventsource server","eventSourceName":"calander","version":"vv1.3.0+7591146.dirty"}
{"level":"info","ts":1619110843.5465922,"logger":"argo-events.eventsource","caller":"metrics/metrics.go:172","msg":"starting metrics server","eventSourceName":"calander"}
{"level":"fatal","ts":1619110843.5519965,"logger":"argo-events.eventsource","caller":"leaderelection/leaderelection.go:111","msg":"failed to new a node","eventSourceName":"calander","error":"open /tmp/graft.log: no such file or directory","stacktrace":"github.com/argoproj/argo-events/common/leaderelection.(*natsEventBusElector).RunOrDie\n\t/home/runner/work/argo-events/argo-events/common/leaderelection/leaderelection.go:111\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).Start\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:293\nmain.main\n\t/home/runner/work/argo-events/argo-events/eventsources/cmd/main.go:65\nruntime.main\n\t/opt/hostedtoolcache/go/1.14.15/x64/src/runtime/proc.go:203"}

[sky29]

@whynowy

Though I tried with another EventSource "Webhook" EventSource ........ here pod is running fine (but no emptyDir volume mounted):

My EventSource file looks like this:

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: webhook
  namespace: argo-events
spec:
  eventBusName: default
  webhook:
    webhook-event:
      port: "12000"
      endpoint: /example
      method: POST

[whynowy]

webhook is expected to work without leaderelecton.

I can not reproduce it from my side. I suspect emptyDir is not supported in k8s 1.15.

[whynowy]

webhook is expected to work without leaderelecton.

I can not reproduce it from my side. I suspect emptyDir is not supported in k8s 1.15.

But you manually patched it, and it worked..... Could you check the head of the eventsource-controller log? Does it say it's version of 1.3.0?

[sky29]

@whynowy

I can not reproduce it from my side. I suspect emptyDir is not supported in k8s 1.15. ---> we are already using emptyDir in other (multiple) kubernetes artifacts, so this should not be a problem (i guess)

here is head of the eventsource-controller log:

kubectl logs eventsource-controller-8dd84fdf5-vk8x2 -n argo-events

{"level":"info","ts":1619102851.8807988,"logger":"argo-events.eventsource-controller","caller":"cmd/main.go:125","msg":"starting eventsource controller","version":"vv1.3.0+7591146.dirty"}

if I am not wrong, it refers argo-event version - v1.3.0

[whynowy]

It is v1.3.0 - that needs to be fixed.

However, I really don't know what's wrong. If it's possible, could you debug by making changes to the source code? The logic of adding emptyDir is at https://github.com/argoproj/argo-events/blob/master/controllers/eventsource/resource.go#L222-L230.

[whynowy]

Found the issue, #1196

[whynowy]

The emptyDir issue is fixed in v1.3.1.