LLM Observability Spans Not Appearing When Payload > 1MB

[lucasiscovici]

Hello,

Context

It seems that when the payload sent to the LLM Observability endpoint (llmobs-intake.datadoghq.eu) exceeds 1MB, the corresponding LLM Obs spans are not correctly ingested. Specifically, we notice that:

• No LLM Obs spans appear in the platform.
• There are no links between APM spans and LLM Obs spans.

In the codebase:

• The dd-trace library sets the max payload size as EVP_PAYLOAD_SIZE_LIMIT = 5 << 20 (i.e. 5MB, with an actual limit of 5.1MB).
• The Datadog Agent configures the EVP proxy with MaxPayloadSize: 5 * 1024 * 1024.

Question

Does the LLM Observability intake endpoint (llmobs-intake.datadoghq.eu) support and process payloads larger than 1MB? If not, is there a recommended way to handle larger payloads for LLM Obs spans?

Thanks in advance!

[lievan]

Hi @lucasiscovici

The SDK should be able to send payloads up to 5MB to llmobs-intake.datadoghq.eu as of ddtrace 3.3.0. There was a bug in previous ddtrace versions where this limit was actually 1MB, so if you are using ddtrace < 3.3.0 I suspect that bug is what's causing your issue.

Note that there is still a per-span size limit of 1MB (e.g. if one LLM generation has over 1mb of text i/o, we will truncate it). We're currently exploring bumping this limit. What per-span size limit would work for your use-case?

Thanks!