Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic error when using otelcol.processor.resourcedetection with ec2 detector #1626

Open
SamLoub opened this issue Sep 6, 2024 · 0 comments
Open

Panic error when using otelcol.processor.resourcedetection with ec2 detector #1626

SamLoub opened this issue Sep 6, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@SamLoub
Copy link

SamLoub commented Sep 6, 2024 •
edited
Loading

What's wrong?

I retrieve the logs from log files. Alloy has full access to those files. I want to add ec2 attributes in those logs with Alloy and send them to otlp endpoint.
Alloy fall in panic error when I configured otelcol.processor.resourcedetection with "ec2" detector. But works fine with other detector such as "system" or "env".
Even when I tail another file or use loki.source.journal as other log source Alloy falls into error.

Steps to reproduce

CONFIG_FILE set to "/etc/alloy/config.alloy"

apt-get install alloy=1.3.1-1   
systemctl start alloy
journalctl -u alloy -f

System information

EC2 OS: Ubuntu 22.04.4 LTS Linux: Linux 6.2.0-1012-aws Architecture: x86_64

Software version

Grafana alloy v1.3.1-1

Configuration

// LOGS
// ------------------------------------------------------------------------------

local.file_match "vaultlog" {
        path_targets = [
                {__path__ = "/var/log/syslog"},
        ]
}

loki.source.file "vaultlog" {
        targets    = local.file_match.vaultlog.targets
        tail_from_end = true
        forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
        output {
                logs = [otelcol.processor.resourcedetection.logs.input]
        }
}

// Add attributes with resource detection
// ------------------------------------------------------------------------------

otelcol.processor.resourcedetection "logs" {
        detectors = ["ec2"]
        timeout = "15s"
        override = false
        output {
                logs = [otelcol.processor.batch.default.input]
        }
}

// OTLPHTTP exporter
// ------------------------------------------------------------------------------

otelcol.processor.batch "default" {
        output {
                logs = [otelcol.exporter.otlphttp.default.input]
        }
}

otelcol.exporter.otlphttp "default" {
        client {
                endpoint = env("ENDPOINT_URL")
        }

        sending_queue {
                enabled       = true
                num_consumers = 10
                queue_size    = 500000
        }

        retry_on_failure {
                enabled          = true
                initial_interval = "500ms"
                max_interval     = "5m"
        }
}

logging {
  level  = "debug"
  format = "logfmt"
}

Logs

Sep 06 13:28:48 ip-10-10-5-36 systemd[1]: alloy.service: Scheduled restart job, restart counter is at 4.
Sep 06 13:28:48 ip-10-10-5-36 systemd[1]: Stopped Vendor-agnostic OpenTelemetry Collector distribution with programmable pipelines.
Sep 06 13:28:48 ip-10-10-5-36 systemd[1]: Started Vendor-agnostic OpenTelemetry Collector distribution with programmable pipelines.
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957632968Z level=info "boringcrypto enabled"=false
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957662039Z level=info msg="running usage stats reporter"
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957668669Z level=info msg="starting complete graph evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.95768635Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=labelstore duration=7.09µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.95769447Z level=info msg="applying non-TLS config to HTTP server" service=http
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.95769932Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=http duration=4.7µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.95770511Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=cluster duration=320ns
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957711581Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=otelcol.exporter.otlphttp.default duration=15.078996ms
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957718071Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=otelcol.processor.batch.default duration=215.066µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957725381Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=otelcol.processor.resourcedetection.logs duration=513.324µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957737171Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=otelcol.receiver.loki.default duration=19.11µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957744652Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=remotecfg duration=29.751µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957753852Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=logging duration=135.484µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957773063Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=otel duration=7.95µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957796963Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=livedebugging duration=13.29µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957814554Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=ui duration=6.86µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.957862995Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=local.file_match.vaultlog duration=37.671µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.958128122Z level=debug msg="no files targets were passed, nothing will be tailed" component_path=/ component_id=loki.source.file.vaultlog
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.958143763Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=loki.source.file.vaultlog duration=264.828µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.958174864Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 node_id=tracing duration=14.611µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.958186314Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=bf65ab4343b6786888d4ee24be1c08c0 duration=16.566987ms
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.958200924Z level=debug msg="changing node state" service=cluster from=viewer to=participant
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.958213325Z level=debug msg="ip-10-10-5-36 @1: participant" service=cluster
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.958447611Z level=info msg="scheduling loaded components and services"
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.958664208Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.processor.resourcedetection.logs duration=136.624µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.958710069Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.receiver.loki.default duration=30.181µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.95875062Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.file.vaultlog duration=28.901µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.958827222Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=otelcol.processor.batch.default duration=61.062µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.959074359Z level=debug msg="tailing new file" component_path=/ component_id=loki.source.file.vaultlog filename=/var/log/syslog
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.95912081Z level=debug msg="finished node evaluation" controller_path=/ controller_id="" node_id=loki.source.file.vaultlog duration=91.862µs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.959165072Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.959191762Z level=debug msg="ip-10-10-5-36 @3: participant" service=cluster
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.959389378Z level=info msg="peers changed" service=cluster peers_count=1 peers=ip-10-10-5-36
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.959533192Z level=debug msg="scheduling components" component_path=/ component_id=otelcol.exporter.otlphttp.default count=3
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.959637415Z level=debug msg="scheduling components" component_path=/ component_id=otelcol.processor.batch.default count=1
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.959656336Z level=debug msg="scheduling components" component_path=/ component_id=otelcol.processor.resourcedetection.logs count=1
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.959702547Z level=info msg="began detecting resource information" component_path=/ component_id=otelcol.processor.resourcedetection.logs
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.960217051Z level=info msg="Seeked /var/log/syslog - &{Offset:587243506 Whence:0}" component_path=/ component_id=loki.source.file.vaultlog
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.960226752Z level=info msg="now listening for http traffic" service=http addr=127.0.0.1:12345
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: ts=2024-09-06T13:28:48.960339755Z level=info msg="tail routine: started" component_path=/ component_id=loki.source.file.vaultlog component=tailer path=/var/log/syslog
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: panic: runtime error: invalid memory address or nil pointer dereference
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5e791e4]
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: goroutine 162 [running]:
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: go.opentelemetry.io/collector/pdata/pcommon.Resource.Attributes(...)
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /go/pkg/mod/go.opentelemetry.io/collector/pdata@v1.9.0/pcommon/generated_resource.go:55
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor/internal.IsEmptyResource(...)
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor@v0.102.0/internal/resourcedetection.go:197
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor/internal.MergeResource({0xc002f455d8?, 0xc0020a432c?}, {0x0?, 0x0?}, 0xc8?)
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor@v0.102.0/internal/resourcedetection.go:179+0x24
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor.(*resourceDetectionProcessor).processLogs(0xc002ffc780, {0x414ca5?, 0x10?}, {0xc002f1a078?, 0xc0020a432c?})
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor@v0.102.0/resourcedetection_processor.go:68+0x67
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: go.opentelemetry.io/collector/processor/processorhelper.NewLogsProcessor.func1({0x92bead0, 0xc003075d10}, {0xc002f1a078?, 0xc0020a432c?})
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /go/pkg/mod/go.opentelemetry.io/collector/processor@v0.102.1/processorhelper/logs.go:48 +0x10b
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: go.opentelemetry.io/collector/consumer.ConsumeLogsFunc.ConsumeLogs(...)
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /go/pkg/mod/go.opentelemetry.io/collector/consumer@v0.102.1/logs.go:25
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: github.com/grafana/alloy/internal/component/otelcol/internal/lazyconsumer.(*Consumer).ConsumeLogs(0xc003084c00, {0x92bead0, 0xc003075d10}, {0xc002f1a060?, 0xc0020a4328?})
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /drone/src/internal/component/otelcol/internal/lazyconsumer/lazyconsumer.go:109 +0x173
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: github.com/grafana/alloy/internal/component/otelcol/receiver/loki.(*Component).Run(0xc003064600, {0x92bead0, 0xc003075d10})
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /drone/src/internal/component/otelcol/receiver/loki/loki.go:92 +0x152
Sep 06 13:28:48 ip-10-10-5-36 systemd[1]: alloy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: github.com/grafana/alloy/internal/runtime/internal/controller.(*BuiltinComponentNode).Run(0xc003062b48, {0x92bead0, 0xc003075d10})
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /drone/src/internal/runtime/internal/controller/node_builtin_component.go:316 +0x111
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: github.com/grafana/alloy/internal/runtime/internal/controller.newTask.func1()
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /drone/src/internal/runtime/internal/controller/scheduler.go:140 +0x6e
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]: created by github.com/grafana/alloy/internal/runtime/internal/controller.newTask in goroutine 131
Sep 06 13:28:48 ip-10-10-5-36 alloy[403973]:         /drone/src/internal/runtime/internal/controller/scheduler.go:137 +0x138
Sep 06 13:28:48 ip-10-10-5-36 systemd[1]: alloy.service: Failed with result 'exit-code'.
Sep 06 13:28:49 ip-10-10-5-36 systemd[1]: alloy.service: Scheduled restart job, restart counter is at 5.
Sep 06 13:28:49 ip-10-10-5-36 systemd[1]: Stopped Vendor-agnostic OpenTelemetry Collector distribution with programmable pipelines.
Sep 06 13:28:49 ip-10-10-5-36 systemd[1]: alloy.service: Start request repeated too quickly.
Sep 06 13:28:49 ip-10-10-5-36 systemd[1]: alloy.service: Failed with result 'exit-code'.
Sep 06 13:28:49 ip-10-10-5-36 systemd[1]: Failed to start Vendor-agnostic OpenTelemetry Collector distribution with programmable pipelines.
@SamLoub SamLoub added the bug Something isn't working label Sep 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@SamLoub