-
Notifications
You must be signed in to change notification settings - Fork 617
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Duplicate Span, sometimes #3275
Comments
Is there a way for you to check if the span IDs are duplicated? |
I can confirm that span and traces id are duplicated. I'm testing this connector sending traces from a Django application that use last otel instrumentation library (0.39) and I receive this:
With previous versions (last time i tried it was 0.26) I didn't have this issue. |
Doing some testing I found that opentelemetry v1.12.0rc2 & v0.32b0 was the first release that includes this bug. |
Hey. I'm also getting a similar issue. I'm seeing duplicate spans. |
This seems really hard to reproduce and the culprit could be in several different components. If someone could provide a full repro with a docker compose yaml setup, and pinned python dependencies, we could definitely investigate. |
this is my docker-compose:
import os
from django.core.wsgi import get_wsgi_application
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from uwsgidecorators import postfork
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'faulty.settings')
application = get_wsgi_application()
@postfork
def init_tracing():
trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(
OTLPSpanExporter(endpoint=os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT'))
))
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(
ConsoleSpanExporter()
))
|
@marctc this example is missing Dockerfiles and django app? A github gist or repo I can clone and just run |
there you go https://github.com/marctc/django-otel-demo |
@marctc you are adding two OTLPSpanExporters to the TracerProvider which is causing duplicates. I added this to your diff --git a/service/faulty/faulty/wsgi.py b/service/faulty/faulty/wsgi.py
index 82eacb9..997d833 100644
--- a/service/faulty/faulty/wsgi.py
+++ b/service/faulty/faulty/wsgi.py
@@ -31,3 +31,7 @@ def init_tracing():
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(
ConsoleSpanExporter()
))
+
+ print(
+ [sp.span_exporter for sp in trace.get_tracer_provider()._active_span_processor._span_processors]
+ ) and it prints out
If you look at the Python startup logs, you'll see the warning You should only use one of these two options for setting up your SDK:
|
@aabmass But I NEVER use opentelemetry-instrument. |
The repro you provided is using it https://github.com/marctc/django-otel-demo/blob/9a3e1d614fda963578ff752db524283562520252/docker-compose.yaml#L7 |
We are Observing the same Issue, sometimes it is 20x, 8x , 2x duplication. Its pretty random and ruining the traces. |
Steps to reproduce
When I use opentelemetry in python3, sometimes I got duplicate span .
these code used to create tracer:
this code used to create span:
this code used to end span:
Something that needs to be said IT IS NOT 100%, when I restart my server It may be disappeared. But Something when I restart my server again, It may be appear.
I swear It's duplicate by report step. cause console print twice already.
In Jaeger It looks like this:
What is the expected behavior?
What happened?
What is the actual behavior?
Additional context
The text was updated successfully, but these errors were encountered: