Monitoring : Pipeline metrics are not generated.

[nagaprabhu]

Problem
Metrics for the pipeline "measurement":"events" are missing. (tremor version: 0.9.3)

My Yaml file has onramp and offramp. I have a trickle file for pipeline.
In the metrics, I am able to see {"measurement":"ramp_events"} ==> This ones specified in the yaml file.
However, the metric for the pipeline is missing ==> The ones specified in a Trickle file.

simple.trickle

#!config metrics_interval_s = 5
select event from in into out;

main.yaml

binding:
  - id: example                                     # The unique name of this binding template
    links:
      '/onramp/metronome/{instance}/out':            # Connect the inpunt to the pipeline
       - '/pipeline/simple/{instance}/in'
      '/pipeline/simple/{instance}/out':           # Connect the pipeline to the output
       - '/offramp/stdout-output/{instance}/in'
      '/pipeline/system::metrics/system/out':
       - '/offramp/stdout-output/{instance}/in'

onramp:
  - id: metronome
    metrics_interval_s: 5
    type: metronome
    config:
      interval: 1000

mapping:
  /binding/example/passthrough:
    instance: "passthrough"

offramp:
  - id: stdout-output
    type: stdout

Log

Dec 10 18:36:36 MyHost tremor[3234]: tremor version: 0.9.3
Dec 10 18:36:36 MyHost tremor[3234]: tremor instance: tremor
Dec 10 18:36:36 MyHost tremor[3234]: rd_kafka version: 0x000000ff, 1.5.0
Dec 10 18:36:36 MyHost tremor[3234]: allocator: snmalloc
Dec 10 18:36:36 MyHost tremor[3234]: Listening at: http://0.0.0.0:9898
Dec 10 18:36:37 MyHost tremor[3234]: {"measurement":"ramp_events","tags":{"ramp":"tremor://localhost/onramp/metronome/passthrough/out","port":"in"},"fields":{"count":0},"timestamp":1607643397487725499}
Dec 10 18:36:37 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643397487722637,"id":0}
Dec 10 18:36:37 MyHost tremor[3234]: {"measurement":"ramp_events","tags":{"ramp":"tremor://localhost/onramp/metronome/passthrough/out","port":"out"},"fields":{"count":0},"timestamp":1607643397487725499}
Dec 10 18:36:37 MyHost tremor[3234]: {"measurement":"ramp_events","tags":{"ramp":"tremor://localhost/onramp/metronome/passthrough/out","port":"error"},"fields":{"count":0},"timestamp":1607643397487725499}
Dec 10 18:36:38 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643398487936747,"id":1}
Dec 10 18:36:39 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643399488079566,"id":2}
Dec 10 18:36:40 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643400488220781,"id":3}
Dec 10 18:36:41 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643401488329222,"id":4}
Dec 10 18:36:42 MyHost tremor[3234]: {"measurement":"ramp_events","tags":{"ramp":"tremor://localhost/onramp/metronome/passthrough/out","port":"in"},"fields":{"count":0},"timestamp":1607643402488608149}
Dec 10 18:36:42 MyHost tremor[3234]: {"measurement":"ramp_events","tags":{"ramp":"tremor://localhost/onramp/metronome/passthrough/out","port":"out"},"fields":{"count":5},"timestamp":1607643402488608149}
Dec 10 18:36:42 MyHost tremor[3234]: {"measurement":"ramp_events","tags":{"ramp":"tremor://localhost/onramp/metronome/passthrough/out","port":"error"},"fields":{"count":0},"timestamp":1607643402488608149}
Dec 10 18:36:42 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643402488604743,"id":5}
Dec 10 18:36:43 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643403488775734,"id":6}
Dec 10 18:36:44 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643404488910850,"id":7}
Dec 10 18:36:45 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643405489120321,"id":8}
Dec 10 18:36:46 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643406489357015,"id":9}
Dec 10 18:36:47 MyHost tremor[3234]: {"measurement":"ramp_events","tags":{"ramp":"tremor://localhost/onramp/metronome/passthrough/out","port":"in"},"fields":{"count":0},"timestamp":1607643407489620944}
Dec 10 18:36:47 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643407489617233,"id":10}
Dec 10 18:36:47 MyHost tremor[3234]: {"measurement":"ramp_events","tags":{"ramp":"tremor://localhost/onramp/metronome/passthrough/out","port":"out"},"fields":{"count":10},"timestamp":1607643407489620944}
Dec 10 18:36:47 MyHost tremor[3234]: {"measurement":"ramp_events","tags":{"ramp":"tremor://localhost/onramp/metronome/passthrough/out","port":"error"},"fields":{"count":0},"timestamp":1607643407489620944}
Dec 10 18:36:48 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643408489830014,"id":11}
Dec 10 18:36:49 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643409490063369,"id":12}
Dec 10 18:36:50 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643410490285042,"id":13}
Dec 10 18:36:51 MyHost tremor[3234]: {"onramp":"metronome","ingest_ns":1607643411490503464,"id":14}

Possible Solution(s)

[anupdhml]

Adding more detail to the issue here: for the simple.trickle case above, the in/out nodes of the pipeline actually don't exist since it's a passthrough pipeline so not seeing the pipeline metrics there is expected. There was a similar ask in #211 -- we can document this behavior as part of the monitoring docs so that it does not trip up our users.

However, the issue is real: even for non-passthrough pipelines, we don't see the metrics. An example pipeline that will reproduce it:

define script process
script
  event
end;

create script process;

select event from in into process;
select event from process into out;

It will be fixed as part of #648. Thanks @nagaprabhu for bringing this to our attention!