Skip to content

awss3exporter support mapping attributes #37858

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Erog38 opened this issue Feb 11, 2025 · 11 comments · Fixed by #39634
Closed

awss3exporter support mapping attributes #37858

Erog38 opened this issue Feb 11, 2025 · 11 comments · Fixed by #39634
Labels
enhancement New feature or request exporter/awss3

Comments

@Erog38
Copy link
Contributor

Erog38 commented Feb 11, 2025

Component(s)

exporter/awss3

Is your feature request related to a problem? Please describe.

When mapping logs to an S3 bucket for cold storage, every tenant / prefix needs to be associated with a unique exporter. This leads to a problem trying to use the routing exporter to route logs from centralized collectors.

Describe the solution you'd like

It would be fantastic if the AWS S3 exporter would support similar functionality as the splunk hec exporter allowing to map configuration values from resource attributes. This way incoming logs can be automatically routed based on incoming attributes. Namely the prefix attribute to start.

Describe alternatives you've considered

We've considered alternatives in order to route to unique exporters, but it would become too troublesome to manage over time.

Additional context

No response

@Erog38 Erog38 added enhancement New feature or request needs triage New item requiring triage labels Feb 11, 2025
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@Erog38
Copy link
Contributor Author

Erog38 commented Feb 11, 2025

I would be open to attempting this work provided it's something that would be accepted.

@jkmdeltek15
Copy link

jkmdeltek15 commented Feb 13, 2025

any updates if this will be worked on? this is a functionality that we also need so that the storage of relevant data should be segregated depending on tenant

a functionality similar to splunk exporter is actually good..

@Erog38
Copy link
Contributor Author

Erog38 commented Feb 13, 2025

I'm hopeful to get some feedback from others as well as the maintainers here to see if it's something worth bringing into this exporter such that it has some amount of continued support 😄

@atoulme
Copy link
Contributor

atoulme commented Feb 15, 2025

I am open to adding this support, but I think you will need to stick around and become a codeowner to continue to maintain it. If that sounds good to you, please let me know and I will assign the issue to you.

@atoulme atoulme removed the needs triage New item requiring triage label Feb 15, 2025
@Erog38
Copy link
Contributor Author

Erog38 commented Feb 19, 2025

Thanks @atoulme! I have a coworker (@yshengli) who's open to taking this on and becoming a codeowner. I'll walk him through the process but will need an additional sponsor to help him become a member if you're open to it.

@Erog38
Copy link
Contributor Author

Erog38 commented Mar 4, 2025

@atoulme, sorry to hit you up again, but I'm hopeful you'd be willing to sponsor @yshengli to be a member and take this on.

@atoulme
Copy link
Contributor

atoulme commented Mar 5, 2025

Your coworker is a tad too light in contributions. You also cannot be a sponsor ; you need approver or maintainer role to sponsor.

I have two sponsors that meet the sponsor requirements listed in the community membership guidelines. Among other requirements, sponsors must be approvers or maintainers of at least one repository in the organization and not both affiliated with the same company

@Erog38
Copy link
Contributor Author

Erog38 commented Mar 5, 2025

Ah apologies, I'd thought that as an approver in contrib, I'd met that requirement.

I'm happy to own this in that case and can bring him on to lend a hand if that's okay with you.

@yshengli
Copy link
Contributor

Hi all, I am glad to work on this feature and will be completing the PR recently. Kindly let me know if you have any thoughts, Thank you!

@Erog38
Copy link
Contributor Author

Erog38 commented Mar 27, 2025

I'll also be here and will submit a PR to be a codeowner tomorrow.

vincentfree pushed a commit to ing-bank/opentelemetry-collector-contrib that referenced this issue May 6, 2025
…butes (open-telemetry#39634)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Introduced the `resource_attrs_to_s3/s3_prefix` configuration field to
enable mapping the S3 bucket prefix to an OpenTelemetry resource
attribute. When this field is set, the S3 prefix is determined by the
specified resource attribute, with `s3uploader/s3_prefix` used as a
fallback if the attribute is missing or empty.
<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes
open-telemetry#37858

<!--Describe what testing was performed and which tests were added.-->
#### Testing
- Added unit tests to verify the functionality of the related code
changes
- Built the OpenTelemetry Collector locally to ensure the component
behaves as intended
<!--Describe the documentation added.-->
#### Documentation
Updated the README to document the new configuration field:

- Described its impact on the component’s behavior
- Clarified how it interacts with the `s3uploader/s3_prefix` setting
- Included an example demonstrating how to use it
<!--Please delete paragraphs that you did not use before submitting.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request exporter/awss3
Projects
None yet
4 participants