Routing by origin - service is not good enough #35320
Comments
dmedinag
added
enhancement
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
jpkrohling
removed
needs triage
Theirs is a similar problem, although the solution proposed here is a bit different it could as well, yes. |
|
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
…nce on composite keys in lb (#36567) Right now, there's a problem at high throughput using the load balancer and the `service.name` resource attribute: The load balancers themself get slow. While it's possible to vertically scale them to a point (e.g. about 100k req/sec), as they get slow they star tot back up traffic and block on requests. Applications then can't write as many spans out, and start dropping spans. This commit seeks to address that by extending the load balancing collector to allow create a composite from attributes that can still keep the load balancing decision "consistent enough" to reduce cardinallity, but still spread the load across ${N} collectors. It doesn't make too many assumptions about how the operators will use this, except that the underlying data (the spans) are unlikely to be complete in all cases, and the key generation is "best effort". This is a deviation from the existing design, in which hard-requires "span.name". == Design Notes === Contributor Skill As a contributor, I'm very much new to the opentelemetry collector, and do not anticipate I will be contributing much except for as needs require to tune the collectors that I am responsible for. Given this, the code may violate certain assumptions that are otherwise "well known". === Required Knowledge The biggest surprise in this code was that despite accepting a slice, the routingIdentifierFromTraces function assumes spans have been processed with the batchpersignal.SplitTraces() function, which appears to ensure taht each "trace" contains only a single span (thus allowing them to be multiplexed effectively) This allows the function to be simplified quite substantially. === Use case The primary use case I am thinking about when writing this work is calculating metrics in the spanmetricsconnector component. Essentially, services drive far too much traffic for a single collector instance to handle, so we need to multiplex it in a way that still allows them to be calculated in a single place (limiting cardinality) but also, spreads the load across ${N} collectors. === Traces only implementation This commit addreses this only for traces, as I only care about traces. The logic can likely be extended easily, however. Fixes #35320 Fixes #33660 --------- Signed-off-by: Juraci Paixão Kröhling <[email protected]> Co-authored-by: Andrew Howden <[email protected]> Co-authored-by: Antoine Toulme <[email protected]> Co-authored-by: Antoine Toulme <[email protected]>
Component(s)
exporter/loadbalancing
Is your feature request related to a problem? Please describe.
There exist two keys for routing spans today:
serviceandtraceID.traceIDis great for tail sampling, whileserviceis the only one producing reliable spanmetrics without insane cardinality on big volumes.Routing spans by
service, however, brings a big problem to the table if the differentservices generating spans produce a very uneven volume. In an environment where a small percentage of services produce >99% of the spans, instances of the otelcol running the spanmetrics connector (or any other trace pipeline receiving spans routed by service) sustain a very asymmetric load, overloading a few instances while others stay idle.Describe the solution you'd like
servicedoes a good job for spanmetrics because it produces metric series issued by a single collector of the layer behind the load balancing collector, easing up aggregation and range functions requiring multiple datapoints. But for this reason so wouldpod_namein a k8s environment, or any other proxy for task instance (where task = application that generates spans).In the context of kubernetes, the IP of the incoming connection uniquely identifies a task instance. My proposal is to add
source_ipas a routing key for signals, which would considerably dilute the problem we have withserviceas routing key by seggregating the load of these big services across as many workers as instances that service has. Since big services are also typically the ones with the most instances in our ecosystem (and others in the industry), while it's not as good as traceID in terms of randomness this is a much more efficient routing key and an effective solution for our problem.Note 1: this solution would also solve this issue to and any other environment where the IP uniquely identifies a task instance.
Note 2: this routing key would be applicable for
spans, but also formetricsandlogs.Describe alternatives you've considered
Ad-hoc pipeline for offenders
Again within a kubernetes environment, we've considered omitting the span traffic of these big services on our central spanmetrics pipeline and deploying otelcols as sidecars with such spanmetrics pipeline.
But this really just hacks the problem, increases our architecture complexity and does not solve the real issue of
servicebeing an inherently bad key for symmetric load.Hack the routing key
Since we have the ability to modify
servicebefore it's used as routing key, we introduced a transformer that changes the value ofserviceto{serviceName}%{ip}, to force the loadbalancer exporter to route by this composed key. Then, on the second layer of collectors, we recover the original value ofserviceby removing the artificial suffix and proceed with the processing. This has mitigated the problem for us for now, we see much more symmetric load in our spanmetrics collectors, but again it's just hacking around a limitation on the exporter.This solution was preferred to us over having a custom adaptation of the
loadbalancingexporteron our side so that we simplify the process of maintaining our collectors.Additional context
No response
The text was updated successfully, but these errors were encountered: