STEF exporter incorrectly double-compresses output #38088
Assignees
Labels
Comments
tigrannajaryan
added
bug
tigrannajaryan
added a commit
to tigrannajaryan/opentelemetry-collector-contrib
that referenced
this issue
Feb 20, 2025
### Description Resolves open-telemetry#38088 Previously the compression option was incorrectly set both for STEF Writer and gRPC Client. This was unnecessary and also did not work if the destination gRPC receiver did not enable zstd compressor. This change correctly removes the zstd compression option from gRPC client and only sets it as a STEF Writer option. ### Testing Done Added zstd compression option to TestExport unit test. Tested manually with STEF Server here https://github.com/splunk/stef/tree/main/otelcol/cmd/stefmockserver Built Collector contrib via `make otelcontribcol`. Used the following config for testing: ```yaml receivers: hostmetrics: collection_interval: 1s scrapers: load: filesystem: memory: network: paging: processes: exporters: debug: stef: endpoint: localhost:4320 compressions: zstd tls: insecure: true processors: service: pipelines: metrics: receivers: [hostmetrics] processors: [] exporters: [debug,stef] ``` Interestingly, with zstd enabled we get less than 1 byte per metric data point with STEF! ``` $ ./stefmockserver_darwin_arm64 2025/02/20 11:28:48.335835 Listening for STEF/gRPC on port 4320 2025/02/20 11:49:55.002889 Incoming STEF/gRPC connection. Records: 1285639, Messages: 4420, Bytes: 1139887, Bytes/point: 0.89, Acks: 4424, Last ACKID: 1285639 ```
tigrannajaryan
added a commit
to tigrannajaryan/opentelemetry-collector-contrib
that referenced
this issue
Feb 20, 2025
### Description Resolves open-telemetry#38088 Previously the compression option was incorrectly set both for STEF Writer and gRPC Client. This was unnecessary and also did not work if the destination gRPC receiver did not enable zstd compressor. This change correctly removes the zstd compression option from gRPC client and only sets it as a STEF Writer option. ### Testing Done Added zstd compression option to TestExport unit test. Tested manually with STEF Server here https://github.com/splunk/stef/tree/main/otelcol/cmd/stefmockserver Built Collector contrib via `make otelcontribcol`. Used the following config for testing: ```yaml receivers: hostmetrics: collection_interval: 1s scrapers: load: filesystem: memory: network: paging: processes: exporters: debug: stef: endpoint: localhost:4320 compressions: zstd tls: insecure: true processors: service: pipelines: metrics: receivers: [hostmetrics] processors: [] exporters: [debug,stef] ``` Interestingly, with zstd enabled we get less than 1 byte per metric data point with STEF! ``` $ ./stefmockserver_darwin_arm64 2025/02/20 11:28:48.335835 Listening for STEF/gRPC on port 4320 2025/02/20 11:49:55.002889 Incoming STEF/gRPC connection. Records: 1285639, Messages: 4420, Bytes: 1139887, Bytes/point: 0.89, Acks: 4424, Last ACKID: 1285639 ```
tigrannajaryan
added a commit
to tigrannajaryan/opentelemetry-collector-contrib
that referenced
this issue
Feb 20, 2025
### Description Resolves open-telemetry#38088 Previously the compression option was incorrectly set both for STEF Writer and gRPC Client. This was unnecessary and also did not work if the destination gRPC receiver did not enable zstd compressor. This change correctly removes the zstd compression option from gRPC client and only sets it as a STEF Writer option. ### Testing Done Added zstd compression option to TestExport unit test. Tested manually with STEF Server here https://github.com/splunk/stef/tree/main/otelcol/cmd/stefmockserver Built Collector contrib via `make otelcontribcol`. Used the following config for testing: ```yaml receivers: hostmetrics: collection_interval: 1s scrapers: load: filesystem: memory: network: paging: processes: exporters: debug: stef: endpoint: localhost:4320 compressions: zstd tls: insecure: true processors: service: pipelines: metrics: receivers: [hostmetrics] processors: [] exporters: [debug,stef] ``` Interestingly, with zstd enabled we get less than 1 byte per metric data point with STEF! ``` $ ./stefmockserver_darwin_arm64 2025/02/20 11:28:48.335835 Listening for STEF/gRPC on port 4320 2025/02/20 11:49:55.002889 Incoming STEF/gRPC connection. Records: 1285639, Messages: 4420, Bytes: 1139887, Bytes/point: 0.89, Acks: 4424, Last ACKID: 1285639 ```
tigrannajaryan
added a commit
to tigrannajaryan/opentelemetry-collector-contrib
that referenced
this issue
Feb 20, 2025
### Description Resolves open-telemetry#38088 Previously the compression option was incorrectly set both for STEF Writer and gRPC Client. This was unnecessary and also did not work if the destination gRPC receiver did not enable zstd compressor. This change correctly removes the zstd compression option from gRPC client and only sets it as a STEF Writer option. ### Testing Done Added zstd compression option to TestExport unit test. Tested manually with STEF Server here https://github.com/splunk/stef/tree/main/otelcol/cmd/stefmockserver Built Collector contrib via `make otelcontribcol`. Used the following config for testing: ```yaml receivers: hostmetrics: collection_interval: 1s scrapers: load: filesystem: memory: network: paging: processes: exporters: debug: stef: endpoint: localhost:4320 compressions: zstd tls: insecure: true processors: service: pipelines: metrics: receivers: [hostmetrics] processors: [] exporters: [debug,stef] ``` Interestingly, with zstd enabled we get less than 1 byte per metric data point with STEF! ``` $ ./stefmockserver_darwin_arm64 2025/02/20 11:28:48.335835 Listening for STEF/gRPC on port 4320 2025/02/20 11:49:55.002889 Incoming STEF/gRPC connection. Records: 1285639, Messages: 4420, Bytes: 1139887, Bytes/point: 0.89, Acks: 4424, Last ACKID: 1285639 ```
tigrannajaryan
added a commit
to tigrannajaryan/opentelemetry-collector-contrib
that referenced
this issue
Feb 20, 2025
### Description Resolves open-telemetry#38088 Previously the compression option was incorrectly set both for STEF Writer and gRPC Client. This was unnecessary and also did not work if the destination gRPC receiver did not enable zstd compressor. This change correctly removes the zstd compression option from gRPC client and only sets it as a STEF Writer option. ### Testing Done Added zstd compression option to TestExport unit test. Tested manually with STEF Server here https://github.com/splunk/stef/tree/main/otelcol/cmd/stefmockserver Built Collector contrib via `make otelcontribcol`. Used the following config for testing: ```yaml receivers: hostmetrics: collection_interval: 1s scrapers: load: filesystem: memory: network: paging: processes: exporters: debug: stef: endpoint: localhost:4320 compressions: zstd tls: insecure: true processors: service: pipelines: metrics: receivers: [hostmetrics] processors: [] exporters: [debug,stef] ``` Interestingly, with zstd enabled we get less than 1 byte per metric data point with STEF! ``` $ ./stefmockserver_darwin_arm64 2025/02/20 11:28:48.335835 Listening for STEF/gRPC on port 4320 2025/02/20 11:49:55.002889 Incoming STEF/gRPC connection. Records: 1285639, Messages: 4420, Bytes: 1139887, Bytes/point: 0.89, Acks: 4424, Last ACKID: 1285639 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component(s)
exporter/stef
What happened?
Description
The zstd compression option is incorrectly set both for STEF Writer and gRPC Client. This is unnecessary and also does not work if the destination gRPC receiver does not enable zstd compressor.
Steps to Reproduce
Use "compression: zstd" option with stef exporter and mock server that does not support zstd gRPC compression.
Expected Result
It should work.
Actual Result
gRPC connection fails.
Collector version
408f780
Environment information
No response
OpenTelemetry Collector configuration
Log output
2025-02-20T11:36:02.314-0500 error internal/queue_sender.go:128 Exporting failed. Dropping data. {"otelcol.component.id": "stef", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", " error": "interrupted due to shutdown: failed to connect to destination: failed to receive from server: rpc error: code = Unimplemented desc = grpc: Decompressor is not installed for grpc-encoding \"zstd\"", "dropp ed_items": 291}Additional context
No response
The text was updated successfully, but these errors were encountered: