[receiver/kafka_metrics_receiver] Provide a way to enable cluster metadata refresh on custom intervals

[shivanshuraj1333]

Component(s)

No response

Is your feature request related to a problem? Please describe.

Say there are multiple kafka brokers running, if a broker goes down, instead of refreshing the metadata immediately the sarama library uses the cached cluster metadata for it.

2025-02-13T10:05:41.990Z	error	[email protected]/obs_metrics.go:61	Error scraping metrics	{"otelcol.component.id": "kafkametrics", "otelcol.component.kind": "Receiver", "otelcol.signal": "metrics", "error": "write tcp 10.244.0.115:47710->10.244.0.120:9092: write: broken pipe"}
go.opentelemetry.io/collector/scraper/scraperhelper.wrapObsMetrics.func1
	/go/pkg/mod/go.opentelemetry.io/collector/scraper/[email protected]/obs_metrics.go:61
go.opentelemetry.io/collector/scraper.ScrapeMetricsFunc.ScrapeMetrics
	/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/metrics.go:24
go.opentelemetry.io/collector/scraper/scraperhelper.scrapeMetrics
	/go/pkg/mod/go.opentelemetry.io/collector/scraper/[email protected]/controller.go:256
go.opentelemetry.io/collector/scraper/scraperhelper.NewMetricsController.func1
	/go/pkg/mod/go.opentelemetry.io/collector/scraper/[email protected]/controller.go:228
go.opentelemetry.io/collector/scraper/scraperhelper.(*controller[...]).startScraping.func1
	/go/pkg/mod/go.opentelemetry.io/collector/scraper/[email protected]/controller.go:175

And also during leader election, there's similar problems

2025-02-13T10:05:43.990Z	error	[email protected]/obs_metrics.go:61 Error scraping metrics  {"kind": "receiver", "name": "kafkametrics", "data_type": "metrics", "error": "kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes; kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes", "scraper": "topics"}

Describe the solution you'd like

Similar problems are seen in issues like:
IBM/sarama#2173
IBM/sarama#2237
IBM/sarama#272

There could be other ways to solve this problem like, but I propose exposing sarama config to let the user configure the RefreshFrequency https://github.com/IBM/sarama/blob/main/config.go#L151

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners for receiver/kafkametrics: @dmitryax. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.