Skip to content

Inconsistent case causes logs dropped drop due to conflict in ES/OS #4133

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
94DanielBrown opened this issue Apr 9, 2025 · 1 comment
Open

Comments

@94DanielBrown
Copy link
Contributor

Bug Description
We get continuously reoccuring alerts that logs are being dropped in Opensearch now for aws load balancer controller

Container aws-load-balancer-controller in namespace kube-system is dropping logs due to type mismatch (MapperParsingException)

Steps to Reproduce
Just run the controller in a busy environment and have alerts for logs being dropped

  • Step-by-step guide to reproduce the bug:
  • Manifests applied while reproducing the issue:
  • Controller logs/error messages while reproducing the issue:

Expected Behavior
Logs aren't dropped

Actual Behavior
Logs are dropped and alert is continuously firing as a result

Regression
Was the functionality working correctly in a previous version ? [Yes / No]
If yes, specify the last version where it worked as expected

Current Workarounds
I have a PR to fix

Environment

  • AWS Load Balancer controller version: {"level":"info","ts":"2025-03-31T05:32:47Z","msg":"version","GitVersion":"v2.12.0","GitCommit":"ab69d951f7f42ce91f5d7880118b8fc7937794f1","BuildDate":"2025-03-10T17:20:40+0000"}
  • Kubernetes version: v1.31.6-eks-bc803b4
  • Using EKS (yes/no), if so version?: yes
  • Using Service or Ingress: service
  • AWS region:
  • How was the aws-load-balancer-controller installed: helm chart via terraform
    • If helm was used then please show output of helm ls -A | grep -i aws-load-balancer-controller
      aws-lb-controller kube-system 3 2025-03-21 15:49:05.046662296 +0000 UTC deployed aws-load-balancer-controller-1.12.0 v2.12.0
      aws-lb-controller-crds kube-system 1 2022-06-14 10:36:53.448558342 +0100 BST deployed lb-controller-crds-0.1.0 0.1.0

Possible Solution (Optional)

I've made a fork and can make a PR, where I found
d.logger.Info("enqueued new deferred TGB", "TGB", nsn.Name)
and
d.logger.Info("TGB not eligible for deferral", "TGB", nsn)
and just changed the TGB field to tgb, these are the two logs I've found examples of it being a field and uppercase

Contribution Intention (Optional)

Yes, I'm willing to submit a PR to fix this issue

@94DanielBrown
Copy link
Contributor Author

Made a PR here:
#4134

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant