Metrics API RFCs to eliminate Raw statistics

[jmacd]

No description provided.

[bogdandrutu]

Sorry, I am a bit lost. Does this represent 4 RFCs? If yes should we split them? Otherwise should we make them one RFC?

[jmacd]

Answered your question here:
open-telemetry/opentelemetry-specification#169

[jmacd]

I think we should review these as a block, and if any need to be removed from this PR, they will be.

[iredelmeier]

Overall LGTM

[bogdandrutu]

I think there is a lot of confusion about this, maybe worth discussing today in the SIG meeting. I will give a 10 minutes presentation about metrics.

[bogdandrutu]

"Raw statistics" which probably is not a good name (I made it up, and probably confused more than helped) is not for "uncommon and vendor-specific aggregations". We use them for example in gRPC and HTTP metrics where we don't want the framework developers (gRPC devs) to define the aggregation and labels for the end user.

[iredelmeier]

@bogdandrutu HTTP and gRPC metrics seem to me like they should be regular metrics/not "raw statistics". Using them in this way seems inline with, for instance, Prometheus practices - see, e.g., the metric and label naming guide, the label examples in their querying examples, and [suggestions around label usage](https://prometheus.io/docs/practices/instrumentation/#things-to-watch-out-for.

Could you clarify why you think that HTTP and gRPC are better served by raw statistics?

[bogdandrutu]

See one of the example I gave to you in the other comment. I think this is what I mentioned in the SIG call that we do have different understanding of the "raw statistics" (delayed metrics). We do not want the gRPC/HTTP devs to decide how the metrics are aggregated, we want them to only record all these measurements and let the application owner decide how these will be aggregated.

One important thing is that the result of the "raw statistics" (delayed metrics) after the "Views" are applied are regular metrics, the main difference being who decides what are the label keys and aggregation function to be used.

[bogdandrutu]

For the "raw statistics" we don't know in advance all the label keys that will be used to break down these metrics. In OpenCensus what we did was to allow users to call record with what we called extra labels. We cannot do the same optimization that we do for the pre-defined metrics (aggregation, labels are pre-defined) because we don't know at compilation time the set of labels that we need to use. Also for "raw statistics" the final set of label values will be a subset of Tags (a.k.a DistributedContext) and the extraLabels.

[iredelmeier]

At least from my understanding, the idea isn't that label values are known at compile time, but that label keys are known. Does that clarify things?

[bogdandrutu]

That is opposite to the idea of the "raw statistics" ("delayed metrics"). The whole point to support this is to allow deferred/delayed label keys and aggregation. See for example https://github.com/census-instrumentation/opencensus-specs/blob/master/stats/gRPC.md we do record all these measurements and recommend some views, but application owners (who configured the Views) can define any label keys they want. One example is when an A/B test is performed in ServiceA (ServiceA calls ServiceB), you would want to add the test_label as a label key for all rpc stats (maybe just some of them) to monitor the effects. But you cannot change the gRPC instrumented code so then we need to allow this mechanism.

[iredelmeier]

That sounds like an issue with the gRPC instrumentation that can be solved with increased configurability, e.g., allowing end users to override the default label names upfront.

[bogdandrutu]

It is not about "not allowing" to override the default label names, is about adding extra labels that are propagated from different services with the context (tags/distributedContext). This is the whole idea. Also I mentioned the aggregation function (for example do I want to build a histogram with these buckets or that buckets, or just build a Sum).

If you try to come up with that "flexible configuration" you will see that you get very close to what we propose in this framework "raw statistics" + views.

[lizthegrey]

+1000 to being able to pre-specify defaults for some tags, rather than having all the tags need to be re-checked at runtime. To be all technical... this feels like the metrics version of performing partial application.

[bogdandrutu]

I agree and also I talked with @SergeyKanzhelev about this. One important optimization for these metrics is "batch" recording. Because we don't know the labels and aggregation in advance what we do is we put every record entry into a producer-consumer queue and the consumer thread "applies" all defined views (views define the aggregation and labels for these metrics so that we can produce a Metric data). Alternative consideration is to do the work on the critical path and extract the necessary labels and apply the aggregation there, based on my experience with this system (used extensively in Google) this has more overhead than the previous implementation.

An option that I had in mind is to have Measure and MeasureCollection where MeasureCollection is a set of Measures that allow batch record for all the Measures in this collection.

[bogdandrutu]

I shared the same concern in my previous question on this PR. I think it makes more sense to have one or two independent RFC than a chain. I don't feel that strong right now during the review but probably when we merge this we can also merge the RFCs into one or two independent once.

[bogdandrutu]

I agree with the motivation see open-telemetry/opentelemetry-specification#145 which proposes to remove the Measurement class and move the Record method on the Measure.

[bogdandrutu]

I think I confused you a bit because we did not define the Histograms/Summaries see open-telemetry/opentelemetry-specification#146.

The idea is the following, we offer support for users to record metrics where the developer that instruments the code can choose between exporting a pre-defined aggregation/label metrics (type A) or a "raw statistics" (type B) (no aggregation, labels pre-defined). The way how this decision is made tries to follow this rules:

1. If the metric is pre-aggregated (e.g. CPU usage is a counter in /proc) and no labels (here I mean that there are no labels used to break down this metric in /proc, but constant labels can be added to this metric if necessary. Then a type A metric should be used because we cannot apply any other aggregation (if you want to calculate rate which is possible for any counter (monotonic increasing metric) that can be done in all modern backends Prometheus, Stackdriver, etc.).
2. If the metric is recorded with every request (rpc latency, bytes received, bytes send) and the developer of the instrumentation is not the application owner (third-party library like OpenTelemetry itself) then type B should be used. This way the application owner can decide how to aggregate these metrics.
3. If the metric is a simple metric that does not make sense to aggregate in a different way (e.g. queue length - this is a gauge and probably does not require any labels to be used to breakdown) then a type A metric should be used.

[bogdandrutu]

We will have a meeting tomorrow (July 9) at 1pm PST. Please use https://lightstep.zoom.us/j/516112682

[bogdandrutu]

One thing that I remember during the meeting about this RFC is how for example gRPC will set the grpc_method. In OpenCensus we decided to set the method as a Tag and the measure will read the value from the tags not from the extra labels provided by gRPC. The main reason is that we want other metrics recorded for this request to be able to use the grpc_method tag as a label as well.

For the status we used the extra labels (the measure labels proposed in this RFC) because this is known at the end of the request and cannot be used in other metrics.

So the main label that a system like gRPC will add which is the request (method) name is read from the Tags (a.k.a. DistributedContext).

[bogdandrutu]

We need to allow users to configure the aggregation for the "raw statistics" as well as the set of labels. In the OpenCensus case the full set of labels supported are Tags (available in the context when the record method is called) + extraLables (specifically recorded with every request), and users should define a subset from there.

[bogdandrutu]

During the meeting there was a mention about CounterVec in Prometheus. Based on my understanding the CounterVec represents what in OpenTelemetry we call Counter and Prometheus Counter represents what we call a TimeSeries.

So we do support the same things for Counters and Gauges.

[jmacd]

I've put together a PR to demonstrate how I would satisfy these RFCs in Go.

jmacd/opentelemetry-go#2

I'd prefer not to drag out the individual threads above. We have a meeting to discuss on Monday.

I think RFC 0001 has good support so far.

RFC 0002 and 0004 created some confusion. I believe the requirement is to allow creation of metrics that do not have a semantic interpretation, to allow SDKs to do what's best and not necessarily to follow the programmer's recommendation. I agree with this goal, and see the main change here as a compromise. Let the programmer recommend a good default at the place where metrics are defined, but call it "advisory". This way the SDK can still implement the behavior it wants, potentially disregarding the programmer's input.

[jmacd]

I will merge this in the "proposed" state, as discussed in the spec meeting today.

[jmacd]

@iredelmeier Actually, I can't merge this. I wasn't sure where to write the current status, so I put it at the top. Will you have a look?

[bhs]

@jmacd what's the current status on this PR? What are we blocked on, if anything? Thanks in advance...

[bogdandrutu]

Some highlevel comments before we can merge this:

1. Use the path suggested in the readme text/XYZ.
2. Consider to merge this RFCs into one.
3. Consider to add details about the MeasureBatch.

[jmacd]

@iredelmeier I've assigned numbers 3, 4, 5, and 6 to these RFCs, which I think can now be submitted in the Status: proposed state as discussed at this morning's otel-metrics meeting.

[bogdandrutu]

Sorry for being probably a bit late into this. I feel that I am not 100% confident on the last RFC, can we split this PR and merge the first 3 which we have no objections and discuss a bit more the 4th one?

[bogdandrutu]

I think based on the discussions we do agree that Measure is different than Summary/Histogram. Measure can produce a Summary or Histogram but can also produce a Counter.

[bogdandrutu]

We decided that we will support a MeasureBatch.

[bogdandrutu]

Don't think you need min/max. Also based on the OpenMetrics and also Prometheus/Stackdriver (it has a max/min but does not use it) they do not have a min/max.

[bogdandrutu]

Please also rebase

[jmacd]

OK. I will merge the first three into one RFC and leave the last one separate. I will post this later today.

[jmacd]

Please take another look @bogdandrutu. I've rebased and merged the first three RFCs into one. The remaining RFCs are number 3 and 4.

[bogdandrutu]

Sorry probably my English was not the best. I meant to merge this PR with the first 3 RFCs (or currently I think we should merge the PR number 3) and open a new PR with RFC 4. I have some questions and concerns about letting users to define the aggregation during the instrumentation.

Does that make sense?

[jmacd]

I thought our goal was to merge these PRs so that individual RFCs can be assigned unique numbers, and then we can discuss them via new PRs against these drafts, which will have Status: proposed. There are two RFCs and we can discuss the now-combined-for-approval RFC 3 independently from the still-needs-discussion RFC 4 about aggregation support.