Skip to content

[opampsupervisor] Panic occurred in agent while configuring the TCP log receiver. #39474

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dhruvit-crest opened this issue Apr 17, 2025 · 12 comments · Fixed by #39485
Closed

[opampsupervisor] Panic occurred in agent while configuring the TCP log receiver. #39474

dhruvit-crest opened this issue Apr 17, 2025 · 12 comments · Fixed by #39485
Assignees
@VihasMakwana
Labels
cmd/opampsupervisor extension/opamp priority:p1 High receiver/tcplog

Comments

@dhruvit-crest
Copy link

Component(s)

cmd/opampsupervisor, cmd/otelcontribcol, receiver/tcplog, extension/opamp

What happened?

Description

When running collector using OpAMP supervisor it's panic on notifyConfig method for tcplog reciever.

Steps to Reproduce

Use the OpAMP Supervisor to connect with the server and attempt to apply the effective configuration to the provided configuration.

Expected Result

Config should be applied and collector should report back effective config.

Actual Result

Collector was in crashloop back with provided error.

Collector version

v0.123.1

Environment information

OS: Ubuntu 24.04

OpenTelemetry Collector configuration

exporters:
    nop: null
extensions:
    health_check:
        endpoint: localhost:35221
    opamp:
        capabilities:
            reports_available_components: false
        instance_uid: 019643a2-9460-7bf8-9de0-43645c31c6c4
        ppid: 3450714
        ppid_poll_interval: 5s
        server:
            ws:
                endpoint: ws://127.0.0.1:38101/v1/opamp
                tls:
                    insecure: true
receivers:
    tcplog:
        add_attributes: true
        listen_address: 0.0.0.0:54525
service:
    extensions:
        - health_check
        - opamp
    pipelines:
        logs/nop:
            exporters:
                - nop
            receivers:
                - tcplog
    telemetry:
        logs:
            encoding: json
        resource:
            host.arch: amd64
            host.name: ub24-50-13-68
            os.description: Ubuntu 24.04
            os.type: linux
            service.instance.id: 019643a2-9460-7bf8-9de0-43645c31c6c4
            service.name: otelcol-contrib
            service.version: 0.123.1

Log output

{"level":"info","ts":"2025-04-17T17:34:18.573+0530","caller":"[email protected]/service.go:197","msg":"Setting up own telemetry..."}
{"level":"info","ts":"2025-04-17T17:34:18.575+0530","caller":"[email protected]/service.go:264","msg":"Starting otelcol-contrib...","Version":"0.123.1","NumCPU":12}
{"level":"info","ts":"2025-04-17T17:34:18.575+0530","caller":"extensions/extensions.go:41","msg":"Starting extensions..."}
{"level":"info","ts":"2025-04-17T17:34:18.575+0530","caller":"extensions/extensions.go:45","msg":"Extension is starting..."}
{"level":"info","ts":"2025-04-17T17:34:18.575+0530","caller":"[email protected]/healthcheckextension.go:32","msg":"Starting health_check extension","config":{"Endpoint":"localhost:35221","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"ReadTimeout":0,"ReadHeaderTimeout":0,"WriteTimeout":0,"IdleTimeout":0,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
{"level":"info","ts":"2025-04-17T17:34:18.575+0530","caller":"extensions/extensions.go:62","msg":"Extension started."}
{"level":"info","ts":"2025-04-17T17:34:18.575+0530","caller":"extensions/extensions.go:45","msg":"Extension is starting..."}
{"level":"info","ts":"2025-04-17T17:34:18.577+0530","caller":"extensions/extensions.go:62","msg":"Extension started."}
panic: cannot marshal type: tcp.SplitFuncBuilder [recovered]
	panic: cannot marshal type: tcp.SplitFuncBuilder

goroutine 1 [running]:
gopkg.in/yaml%2ev3.handleErr(0xc000ef48e0)
	gopkg.in/[email protected]/yaml.go:294 +0x68
panic({0x9930c60?, 0xc001a690a0?})
	runtime/panic.go:792 +0x132
gopkg.in/yaml%2ev3.(*encoder).marshal(0xc000981408, {0x0, 0x0}, {0xa08e2c0?, 0x0?, 0xc000ef39a0?})
	gopkg.in/[email protected]/encode.go:182 +0x9d9
gopkg.in/yaml%2ev3.(*encoder).marshal(0xc000981408, {0x0, 0x0}, {0x9dba4c0?, 0xc001a69090?, 0x98?})
	gopkg.in/[email protected]/encode.go:162 +0x74b
gopkg.in/yaml%2ev3.(*encoder).marshal.(*encoder).mapv.func1()
	gopkg.in/[email protected]/encode.go:192 +0xe5
gopkg.in/yaml%2ev3.(*encoder).mappingv(0xc000981408, {0x0?, 0x0}, 0xc000ef3dd0)
	gopkg.in/[email protected]/encode.go:265 +0x14a
gopkg.in/yaml%2ev3.(*encoder).mapv(...)
	gopkg.in/[email protected]/encode.go:187
gopkg.in/yaml%2ev3.(*encoder).marshal(0xc000981408, {0x0, 0x0}, {0xa3a7b80?, 0xc001a392c0?, 0xc000ef3ea0?})
	gopkg.in/[email protected]/encode.go:164 +0x7bf
gopkg.in/yaml%2ev3.(*encoder).marshal(0xc000981408, {0x0, 0x0}, {0x9dba4c0?, 0xc001a68d70?, 0x98?})
	gopkg.in/[email protected]/encode.go:162 +0x74b
gopkg.in/yaml%2ev3.(*encoder).marshal.(*encoder).mapv.func1()
	gopkg.in/[email protected]/encode.go:192 +0xe5
gopkg.in/yaml%2ev3.(*encoder).mappingv(0xc000981408, {0x0?, 0x0}, 0xc000ef42d0)
	gopkg.in/[email protected]/encode.go:265 +0x14a
gopkg.in/yaml%2ev3.(*encoder).mapv(...)
	gopkg.in/[email protected]/encode.go:187
gopkg.in/yaml%2ev3.(*encoder).marshal(0xc000981408, {0x0, 0x0}, {0xa3a7b80?, 0xc001a39140?, 0xc000ef43a0?})
	gopkg.in/[email protected]/encode.go:164 +0x7bf
gopkg.in/yaml%2ev3.(*encoder).marshal(0xc000981408, {0x0, 0x0}, {0x9dba4c0?, 0xc001a68d40?, 0x98?})
	gopkg.in/[email protected]/encode.go:162 +0x74b
gopkg.in/yaml%2ev3.(*encoder).marshal.(*encoder).mapv.func1()
	gopkg.in/[email protected]/encode.go:192 +0xe5
gopkg.in/yaml%2ev3.(*encoder).mappingv(0xc000981408, {0x0?, 0x0}, 0xc000ef47d0)
	gopkg.in/[email protected]/encode.go:265 +0x14a
gopkg.in/yaml%2ev3.(*encoder).mapv(...)
	gopkg.in/[email protected]/encode.go:187
gopkg.in/yaml%2ev3.(*encoder).marshal(0xc000981408, {0x0, 0x0}, {0xa3a7b80?, 0xc001a389c0?, 0xc000ef4868?})
	gopkg.in/[email protected]/encode.go:164 +0x7bf
gopkg.in/yaml%2ev3.(*encoder).marshalDoc(0xc000981408, {0x0, 0x0}, {0xa3a7b80?, 0xc001a389c0?, 0x0?})
	gopkg.in/[email protected]/encode.go:105 +0x12e
gopkg.in/yaml%2ev3.Marshal({0xa3a7b80, 0xc001a389c0})
	gopkg.in/[email protected]/yaml.go:222 +0x316
github.com/open-telemetry/opentelemetry-collector-contrib/extension/opampextension.(*opampAgent).composeEffectiveConfig(0xc00175e4b0)
	github.com/open-telemetry/opentelemetry-collector-contrib/extension/[email protected]/opamp_agent.go:413 +0xca
github.com/open-telemetry/opentelemetry-collector-contrib/extension/opampextension.(*opampAgent).Start.func5({0xc39fd8?, 0xc001a1e990?})
	github.com/open-telemetry/opentelemetry-collector-contrib/extension/[email protected]/opamp_agent.go:144 +0x17
github.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).UpdateEffectiveConfig(0xc00187c000, {0xcaf6f10?, 0x13f42dc0?})
	github.com/open-telemetry/[email protected]/client/internal/clientcommon.go:302 +0x3d
github.com/open-telemetry/opamp-go/client.(*wsClient).UpdateEffectiveConfig(0xc00175e4b0?, {0xcaf6f10?, 0x13f42dc0?})
	github.com/open-telemetry/[email protected]/client/wsclient.go:137 +0x1f
github.com/open-telemetry/opentelemetry-collector-contrib/extension/opampextension.(*opampAgent).NotifyConfig(0xc00175e4b0, {0xcaf6f10, 0x13f42dc0}, 0x108?)
	github.com/open-telemetry/opentelemetry-collector-contrib/extension/[email protected]/opamp_agent.go:224 +0x5f
go.opentelemetry.io/collector/service/extensions.(*Extensions).NotifyConfig(0xc00182f700, {0xcaf6f10, 0x13f42dc0}, 0xc001841ac0)
	go.opentelemetry.io/collector/[email protected]/extensions/extensions.go:125 +0x14b
go.opentelemetry.io/collector/service.(*Service).Start(0xc00182cf00, {0xcaf6f10, 0x13f42dc0})
	go.opentelemetry.io/collector/[email protected]/service.go:274 +0x2ae
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents(0xc000803680, {0xcaf6f10, 0x13f42dc0})
	go.opentelemetry.io/collector/[email protected]/collector.go:242 +0xb5d
go.opentelemetry.io/collector/otelcol.(*Collector).Run(0xc000803680, {0xcaf6f10, 0x13f42dc0})
	go.opentelemetry.io/collector/[email protected]/collector.go:296 +0x55
go.opentelemetry.io/collector/otelcol.NewCommand.func1(0xc001756008, {0xb7f3064?, 0x7?, 0xb7e703b?})
	go.opentelemetry.io/collector/[email protected]/command.go:39 +0xa7
github.com/spf13/cobra.(*Command).execute(0xc001756008, {0xc0001740d0, 0x2, 0x2})
	github.com/spf13/[email protected]/command.go:1015 +0xa94
github.com/spf13/cobra.(*Command).ExecuteC(0xc001756008)
	github.com/spf13/[email protected]/command.go:1148 +0x40c
github.com/spf13/cobra.(*Command).Execute(0xbc553b8?)
	github.com/spf13/[email protected]/command.go:1071 +0x13
main.runInteractive({0xbc553b8, {{0xb8351ce, 0xf}, {0xb900e5b, 0x1f}, {0xb7f2eb2, 0x7}}, 0x0, {{{0x0, 0x0, ...}, ...}}, ...})
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:67 +0x5d
main.run(...)
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:10
main.main()
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:60 +0x4dd

Additional context

No response
@dhruvit-crest dhruvit-crest added bug Something isn't working needs triage New item requiring triage labels Apr 17, 2025
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@dhruvit-crest dhruvit-crest mentioned this issue Apr 17, 2025
Closed
@VihasMakwana VihasMakwana mentioned this issue Apr 17, 2025
Merged
@VihasMakwana
Copy link
Contributor

I was able to reproduce this panic. Here's the fix #39485

@VihasMakwana VihasMakwana self-assigned this Apr 17, 2025
@VihasMakwana VihasMakwana added priority:p0 Critical priority:p1 High and removed bug Something isn't working priority:p0 Critical labels Apr 17, 2025
@VihasMakwana
Copy link
Contributor

@open-telemetry/collector-contrib-maintainers I'm marking this as a p1 as it's a panic. Please correct it if I'm mistaken.

@andrzej-stencel andrzej-stencel removed the needs triage New item requiring triage label Apr 18, 2025
@TylerHelmuth
Copy link
Member

@VihasMakwana can you confirm if this is only when using the opampsupervisor? I suspect the opampsupervisor is unrelated and the issue is in the component, which would affect its use anywhere

@VihasMakwana
Copy link
Contributor

@VihasMakwana can you confirm if this is only when using the opampsupervisor? I suspect the opampsupervisor is unrelated and the issue is in the component, which would affect its use anywhere

Sure. I'll confirm that if can trigger panic from elsewhere.

@TylerHelmuth
Copy link
Member

I am specifically curious if the contrib distro panics when this component is configured

@andrzej-stencel
Copy link
Member

I am specifically curious if the contrib distro panics when this component is configured

No, it doesn't. Running this works just fine:

service:
  pipelines:
    logs:
      receivers:
        - tcplog
      exporters:
        - debug/basic
receivers:
  tcplog:
    listen_address: 0.0.0.0:1234
exporters:
  debug/basic:
$ otelcol-contrib-0.124.1-linux_amd64 --config 0418-tcplog-receiver.yaml
2025-04-18T23:08:14.790+0200    info    [email protected]/service.go:199 Setting up own telemetry...
2025-04-18T23:08:14.790+0200    info    builders/builders.go:26 Development component. May change in the future.
2025-04-18T23:08:14.793+0200    info    [email protected]/service.go:266 Starting otelcol-contrib...     {"Version": "0.124.1", "NumCPU": 20}
2025-04-18T23:08:14.793+0200    info    extensions/extensions.go:41     Starting extensions...
2025-04-18T23:08:14.793+0200    info    adapter/receiver.go:41  Starting stanza receiver
2025-04-18T23:08:14.794+0200    info    [email protected]/service.go:289 Everything is ready. Begin running and processing data.

@VihasMakwana
Copy link
Contributor

@TylerHelmuth I also checked other areas that might potentially trigger a panic, but everything seems to be working fine (except running it with opampextension).

The panic triggers at this step

opentelemetry-collector-contrib/extension/opampextension/opamp_agent.go

Lines 412 to 413 in 990e1dc

m := o.effectiveConfig.ToStringMap()
conf, err := yaml.Marshal(m)

The exported SplitFuncBuilder is a part of config map.

opentelemetry-collector-contrib/pkg/stanza/operator/input/tcp/config.go

Lines 65 to 75 in 990e1dc

type BaseConfig struct {
MaxLogSize helper.ByteSize `mapstructure:"max_log_size,omitempty"`
ListenAddress string `mapstructure:"listen_address,omitempty"`
TLS *configtls.ServerConfig `mapstructure:"tls,omitempty"`
AddAttributes bool `mapstructure:"add_attributes,omitempty"`
OneLogPerPacket bool `mapstructure:"one_log_per_packet,omitempty"`
Encoding string `mapstructure:"encoding,omitempty"`
SplitConfig split.Config `mapstructure:"multiline,omitempty"`
TrimConfig trim.Config `mapstructure:",squash"`
SplitFuncBuilder SplitFuncBuilder
}

We then try to marshal it to yaml and it panics.

I suspect the opampsupervisor is unrelated and the issue is in the component, which would affect its use anywhere

yeah, component should exclude such fields (or mark them unexported).

@dhruvit-crest
Copy link
Author

I’ve recently started exploring OpenTelemetry, and I think it would be beneficial to have a unit test or verification method in place to ensure that the all component configurations are properly marshable to YAML (common method if possible). This would help maintain consistency. As Vishal mentioned, the current issue isn’t reproducible with existing configs, but there’s a chance that new components might not follow the same pattern.

atoulme pushed a commit that referenced this issue Apr 21, 2025
@VihasMakwana
[receiver/tcplog] remove SplitFuncBuilder from mapstructure (#39485)
2d91302 
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

The `SplitFuncBuilder` field is part of the collector's effective
configuration, which it shouldn't be. We can't marshal/unmarshal
functions. This PR removes it.
It also solves a collector panic reported by
#39474

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes
#39474

<!--Please delete paragraphs that you did not use before submitting.-->
@tigrannajaryan
Copy link
Member

Can we add auto-generated tests to all components to verify that the component config structure is marshallable?

@VihasMakwana
Copy link
Contributor

Can we add auto-generated tests to all components to verify that the component config structure is marshallable?

Yeah, that was on my mind as well. It should be doable.

We can rely on CreateDefaultConfig() and make sure we can marshal to json/yaml.

@VihasMakwana
Copy link
Contributor

VihasMakwana commented Apr 21, 2025 •
edited
Loading

I'll open up a follow-up issue.

@github-actions github-actions bot mentioned this issue Apr 22, 2025
Closed
@VihasMakwana VihasMakwana mentioned this issue Apr 22, 2025
Open
akshays-19 pushed a commit to akshays-19/opentelemetry-collector-contrib that referenced this issue Apr 23, 2025
@VihasMakwana @AkshayS198
[receiver/tcplog] remove SplitFuncBuilder from mapstructure (open-t…
0157d7b 
…elemetry#39485)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

The `SplitFuncBuilder` field is part of the collector's effective
configuration, which it shouldn't be. We can't marshal/unmarshal
functions. This PR removes it.
It also solves a collector panic reported by
open-telemetry#39474

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes
open-telemetry#39474

<!--Please delete paragraphs that you did not use before submitting.-->
Fiery-Fenix pushed a commit to Fiery-Fenix/opentelemetry-collector-contrib that referenced this issue Apr 24, 2025
@VihasMakwana @Fiery-Fenix
[receiver/tcplog] remove SplitFuncBuilder from mapstructure (open-t…
3ef620a 
…elemetry#39485)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

The `SplitFuncBuilder` field is part of the collector's effective
configuration, which it shouldn't be. We can't marshal/unmarshal
functions. This PR removes it.
It also solves a collector panic reported by
open-telemetry#39474

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes
open-telemetry#39474

<!--Please delete paragraphs that you did not use before submitting.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@VihasMakwana VihasMakwana

Labels
cmd/opampsupervisor extension/opamp priority:p1 High receiver/tcplog
Projects
None yet
Milestone
No milestone
5 participants
@tigrannajaryan @TylerHelmuth @andrzej-stencel @VihasMakwana @dhruvit-crest