Using OTLPTraceExporter and OTLPMetricExporter together causes empty data

[EddieEldridge]

What version of OpenTelemetry are you using?

   "@opentelemetry/api": "^1.0.4",
    "@opentelemetry/api-metrics": "^0.27.0",
    "@opentelemetry/auto-instrumentations-node": "^0.27.3",
    "@opentelemetry/exporter-metrics-otlp-proto": "^0.27.0",
    "@opentelemetry/exporter-trace-otlp-proto": "^0.27.0",
    "@opentelemetry/instrumentation-express": "^0.27.1",
    "@opentelemetry/instrumentation-http": "^0.27.0",
    "@opentelemetry/sdk-node": "^0.27.0",
    "@opentelemetry/sdk-trace-base": "^1.0.1",
    "@opentelemetry/sdk-trace-node": "^1.0.1",
    "@opentelemetry/semantic-conventions": "^1.0.1"

What version of Node are you using?

v12.22.1

Please provide the code you used to setup the OpenTelemetry SDK

// Config
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
import { Resource } from '@opentelemetry/resources'
import { v4 as uuidv4 } from 'uuid';

// Metrics
import { ConsoleMetricExporter, MeterProvider } from '@opentelemetry/sdk-metrics-base';
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto";

// Traces
import { BasicTracerProvider, BatchSpanProcessor, SimpleSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'

const OpenTelemetry = require("@opentelemetry/sdk-node");

// OpenTelemetry Exporter Config
export const exporterConfig = {
  headers: {
    'api-key': process.env.NR_TOKEN,
    'Content-Type': 'application/x-protobuf'
  }
};

export const OTEL_SERVICE_RESOURCE = new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'otel-express-node',
  [SemanticResourceAttributes.SERVICE_VERSION]: '1.0.0',
  [SemanticResourceAttributes.SERVICE_INSTANCE_ID]: uuidv4()
});

// Create our OTLP exporters
const traceExporter = new OTLPTraceExporter(exporterConfig);
const metricExporter = new OTLPMetricExporter(exporterConfig); // <- Adding this line causes Traces to not be ingested correctly

export function registerTraceProvider() {
    const traceProvider = new BasicTracerProvider({
        resource: OTEL_SERVICE_RESOURCE
    })

    traceProvider.addSpanProcessor(new SimpleSpanProcessor(traceExporter));
    traceProvider.register();
    return traceProvider;
}

export async function otelSetupInstrumentation() {

    // 1. Create and register a Trace Provider
    registerTraceProvider();

    // 2. Create an instance of the OpenTelemetry SDK, passing in our exporter and Resource config
    const sdk = new OpenTelemetry.NodeSDK({
        resource: OTEL_SERVICE_RESOURCE,
        traceExporter: traceExporter,
        // metricExporter: metricExporter,
        instrumentations: [getNodeAutoInstrumentations()]
    });

    sdk.start()
}

What did you do?

Tried to send OTLP trace data to NewRelic.

What did you expect to see?

My trace data to be created and ingested correctly.

What did you see instead?

The following error: One or more OTLP payload(s) was dropped because it was empty.

Additional context

If I remove the line
const metricExporter = new OTLPMetricExporter(exporterConfig);
everything works fine. My traces are ingested correctly and displayed in the UI.

I am also able to send metric data fine, just not when I try and do both at the same time.

I am confused why simply instantiating an instance of the OTLPMetricExporter would cause my traces to not be sent/formatted properly even though I don't even pass it into the SDK.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue was closed because it has been stale for 14 days with no activity.

[asim47]

did you find any solution? I have the same problem

[EddieEldridge]

did you find any solution? I have the same problem

No, sorry.

[asim47]

did you find any solution? I have the same problem

No, sorry.

I found a solution, Adding OTPL urls for both tracing and metrics fixed the issue.
I had to create separate exporterConfig for both tracing and metrics:

const headers = {
    'api-key': process.env.NR_TOKEN,
    'Content-Type': 'application/x-protobuf'
  }

export const exporterConfigForTracing = {
  url:   process.env.OTPL_TRACE_URL
  headers, 
};


export const exporterConfigForMetrics = {
  url:   process.env.OTPL_METRICS_URL
  headers,
};

then use it like:

const traceExporter = new OTLPTraceExporter(exporterConfigForTracing);

const metricExporter = new OTLPMetricExporter(exporterConfigForMetrics);

Note* make sure the OTPL urls are in the same region as to where you store your new relic data.

[dyladan]

Can you confirm if this is still happening with the latest version of the proto exporter? I believe this may have been caused by an inheritance issue. The otlp metrics exporters used to depend on the trace exporters as there was some shared transformation code. The code has since been factored out into its own package and the inheritance structure has been simplified

[icco]

I am still seeing this.

packages:

    "@opentelemetry/api": "1.1.0",
    "@opentelemetry/api-metrics": "0.29.2",
    "@opentelemetry/context-async-hooks": ">=1.0.0",
    "@opentelemetry/core": "1.3.1",
    "@opentelemetry/exporter-metrics-otlp-proto": "0.29.2",
    "@opentelemetry/exporter-trace-otlp-proto": "0.29.2",
    "@opentelemetry/instrumentation": "0.29.2",
    "@opentelemetry/instrumentation-aws-lambda": "0.32.0",
    "@opentelemetry/instrumentation-aws-sdk": "0.8.0",
    "@opentelemetry/instrumentation-express": "0.30.0",
    "@opentelemetry/instrumentation-http": "0.29.2",
    "@opentelemetry/instrumentation-mongodb": "0.31.0",
    "@opentelemetry/instrumentation-nestjs-core": "0.30.0",
    "@opentelemetry/propagator-aws-xray": ">=1.0.0 <1.2.0",
    "@opentelemetry/propagator-b3": "1.3.1",
    "@opentelemetry/resources": ">=1.0.0",
    "@opentelemetry/sdk-metrics-base": "0.29.2",
    "@opentelemetry/sdk-node": "0.29.2",
    "@opentelemetry/sdk-trace-base": ">=1.0.0",
    "@opentelemetry/semantic-conventions": ">=1.0.0",

code for initializing exporters:

  let textMapPropagator: TextMapPropagator | undefined;
  let spanProcessor: SpanProcessor | undefined;
  let traceExporter: SpanExporter | undefined;
  if (options.sumoTraceURL || options.OTLP?.traces) {
    traceExporter = new OTLPTraceExporter({
      // Explicitly || incase sumoTraceURL is empty string.
      url: options.sumoTraceURL || options.OTLP?.url + "/v1/traces",
    });
    spanProcessor = new BatchSpanProcessor(traceExporter);
    textMapPropagator = new CompositePropagator({
      propagators: [
        new W3CTraceContextPropagator(),
        new W3CBaggagePropagator(),
        new AWSXRayPropagator(),
        new B3Propagator(),
        new B3Propagator({ injectEncoding: B3InjectEncoding.MULTI_HEADER }),
      ],
    });
  }

  // If there is a sumo metrics url, enable metrics
  let metricReader: MetricReader | undefined;
  if (options.sumoMetricsURL || options.OTLP?.metrics) {
    const metricExporter = new OTLPMetricExporter({
      // Explicitly || incase sumoMetricsURL is empty string.
      url: options.sumoMetricsURL || options.OTLP?.url + "/v1/metrics",
    });
    metricReader = new PeriodicExportingMetricReader({
      exporter: metricExporter,
      exportIntervalMillis: 1000,
    });
  }

[icco]

we're also using NodeSDK, I wonder if the bug could be in there?

[icco]

Nope verified what @EddieEldridge is seeing, its literrally the creation of a OTLPMetricExporter that causes the issue. You don't even need to use it to cause traces to break.

[icco]

Also, it's order dependent.

If

    traceExporter = new OTLPTraceExporter({
      url: "http://localhost:8000/v1/traces",
    });
    const metricExporter = new OTLPMetricExporter({
      url: "http://localhost:8000/v1/metrics",
    });

then trace requests will have empty proto and metrics will have data.

If

    const metricExporter = new OTLPMetricExporter({
      url: "http://localhost:8000/v1/metrics",
    });
    traceExporter = new OTLPTraceExporter({
      url: "http://localhost:8000/v1/traces",
    });

then metric requests will have empty proto and traces will have data.

Also it's not domain dependent. I tried sending exporters to different collectors on different domains and that didn't change anything.

[icco]

Also I tried on both Node v14 and v16 and saw no change.

[dyladan]

Marking p2 since this is not crashing the app but is causing telemetry not to be emitted. I'm looking into this.

[dyladan]

This is fixed by #3098

The root cause was that the ExportRequestProto was a module-level global in a module which was a base class for the metric and trace exporters (old source). @legendecas fixed this while converting the proto exporters to use precompiled JS instead of loading protos in #3098 (link to PR lines) (new source).

[dyladan]

The proposed release fixes this issue.

[EddieEldridge]

The proposed release fixes this issue.

Awesome. Thanks @dyladan and @legendecas. Was questioning my sanity while dealing with this one so I appreciate someone taking a look at it and figuring it out!

[dyladan]

I've also been pulling my hair out on this one including going on a wild goose chase with the gRPC exporters for a couple weeks (my brain skipped right over the fact this is the proto exporters😩). Imagine my surprise when I tracked the issue down finally in my outdated local copy, then a git pull fixed it in front of my eyes 😂)