Extend the support for partition_logs_by_resource_attributes to raw encoding

[namco1992]

Component(s)

exporter/kafka

Is your feature request related to a problem? Please describe.

We've migrated our logging pipeline partially to Otel. We used to have a ELK stack, and now we still persist the logstash that receives logs from a Kafka topic, and insert them to ElasticSearch. On the producer side, we have an Otel collector with Kafka exporter. To keep the backward compatibility, we use the raw encoding in the Kafka exporter, but it doesn't support partition_logs_by_resource_attributes:

opentelemetry-collector-contrib/exporter/kafkaexporter/marshaler.go

Lines 90 to 98 in 0e087ff

	switch encoding {
	case defaultEncoding:
	return newPdataLogsMarshaler(&plog.ProtoMarshaler{}, defaultEncoding, partitionLogsByAttributes), nil
	case "otlp_json":
	return newPdataLogsMarshaler(&plog.JSONMarshaler{}, "otlp_json", partitionLogsByAttributes), nil
	case raw.Encoding():
	return raw, nil
	default:
	return nil, errUnrecognizedEncoding

I don't see why the raw encoding can't do the same thing as other encodings. The raw marshaler indeed discards the resource and record attributes in the end result (i.e. the message sent to the Kafka), but it still has the access to attributes during marshalling, just as other marshalers.

Describe the solution you'd like

The raw encoding should support partition_logs_by_resource_attributes too. Also if #38484 is accepted, raw encoding should also support it.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy @axw

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[axw]

Thanks for opening this, I actually have a WIP branch that fix it, as I'm working on moving the partitioning logic out of the individual marshalers.

[axw]

#39001 should fix this: partitioning will be done outside of encoders with the exception of Jaeger, which does its own thing.