Add Logs support to mdatagen

[cyrusjk]

Component(s)

mdatagen

Describe the issue you're reporting

mdatagen supports metrics and telemetry, but not logs, even though the collector supports log collection. Having wrappers generated that support all of the Open Telemetry types would be nice thing.

[bogdandrutu]

How would you use it? And what for?

[cyrusjk]

In the context of RDBMS receivers, I would like to be able to update the receiver to support the exporting of SQL query and query plan text data to the Logs pipeline along with some attributes such as timestamp, db/schema, user, etc. While Logs scraping can be added to a Receiver, the support is inconsistent and having mdatagen support would help set a standard for how this kind of integration can be done.

[github-actions]

Pinging code owners for cmd/mdatagen: @dmitryax. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[sincejune]

While working on the SQL Server receiver, I encountered similar use cases. I believe there are several benefits, particularly for scraper-based receivers, from supporting logs in mdatagen:

Automatic Generation of Scope Name and Version

Currently, we manually attach the scope name and version when constructing plog. Although this approach works, many scope versions may become outdated over time.

Template-Based Documentation

Documentation can become inconsistent across different scraper-based receivers. Using templates can help maintain consistency and uniformity.

Avoiding Field Type Conflicts

There might be multiple scrapers calling databases and sending different log records for various use cases (e.g., sample query and top query). Field types may unexpectedly differ among these log records, potentially causing issues for subsequent consumers. Having a defined schema will help avoid such conflicts.

Implementation Plan

I plan to add the following specification to the schema for supporting logs in the initial version:

logs:
  sample_log:
    enabled: true
    description: <description>
    extended_documentation: <extended_documentation>
    attributes: [attribute1, attribute2] # reuse the attributes section

We can consider adding configurations as well in the next iteration:

logs:
  sample_log:
    enabled: true
    description: <description>
    extended_documentation: <extended_documentation>
    attributes: [attribute1, attribute2] # reuse the attributes section
    configs:
      lookback_window:
        type: int
        unit: s
      ...

@bogdandrutu I'm willing to implement this if the team agrees with this proposal. cc @open-telemetry/collector-approvers

[dmitryax]

I'm not against this approach. However, we are now introducing some framework for receivers emitting structured log records while most of the receivers wouldn't be able to follow it. So maybe we can start with some generic functionality like Automatic Generation of Scope Name and Version and Template-Based Documentation and apply it to all log receivers. Then we proceed with other features and clearly mark receivers emitting structured logs when some specific features are applied to them. I'm suggesting this approach because it'll bring consistent usage of the mdatagen in log receivers instead of having this applied only to a couple of receivers while others are not even mentioned in metadata.yaml.

[cyrusjk]

Is there a discussion on the "structured log event" or other documentation? It sound similar to other internal work I have seen and I'd like to understand it better, regardless of state or maturity.

…

On Thu, Mar 13, 2025 at 2:52 AM Dmitrii Anoshin ***@***.***> wrote: I'm not against this approach. However, we are now introducing some framework for receivers emitting structured log records while most of the receivers wouldn't be able to follow it. So maybe we can start with some generic functionality like *Automatic Generation of Scope Name and Version* and *Template-Based Documentation* and apply it to all log receivers. Then we proceed with other features and clearly mark receivers emitting structured logs when some specific features are applied to them. I'm suggesting this approach because it'll bring consistent usage of the mdatagen in log receivers instead of having this applied only to a couple of receivers while others are not even mentioned in metadata.yaml. — Reply to this email directly, view it on GitHub <#12571 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEJHBAJ7FNRGY6OTRQCGQAL2UDXHDAVCNFSM6AAAAABYOWLD4SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZGY4TCMJSGQ> . You are receiving this because you authored the thread.Message ID: ***@***.*** com> [image: dmitryax]*dmitryax* left a comment (open-telemetry/opentelemetry-collector#12571) <#12571 (comment)> I'm not against this approach. However, we are now introducing some framework for receivers emitting structured log records while most of the receivers wouldn't be able to follow it. So maybe we can start with some generic functionality like *Automatic Generation of Scope Name and Version* and *Template-Based Documentation* and apply it to all log receivers. Then we proceed with other features and clearly mark receivers emitting structured logs when some specific features are applied to them. I'm suggesting this approach because it'll bring consistent usage of the mdatagen in log receivers instead of having this applied only to a couple of receivers while others are not even mentioned in metadata.yaml. — Reply to this email directly, view it on GitHub <#12571 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEJHBAJ7FNRGY6OTRQCGQAL2UDXHDAVCNFSM6AAAAABYOWLD4SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMJZGY4TCMJSGQ> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>