Unable to Get Internal Metrics for Logs Pipeline via Prometheus Exporter

[shahparthiv]

Hi team,

I'm working with the OpenTelemetry Collector and using a pipeline for processing logs.

Setup:

• Receiver: NetFlow receiver (emits logs)
• Processor: Custom processor that reads and transforms logs to metrics
• Pipeline type: logs
• Exporter: debug

Problem:
I’m expecting internal metrics such as:

• otelcol_receiver_accepted_log_records
• otelcol_processor_incoming_items
• otelcol_processor_outgoing_items

…to be exposed via the Prometheus exporter.

However, even after configuring the Prometheus exporter, these internal metrics are not visible at /metrics.

Question:
What’s the correct way to expose internal pipeline metrics for a logs pipeline using the Prometheus exporter?

Do I need to:

• Add something specific in the pipeline config?
• Enable internal telemetry or metrics level somewhere?
• Use a special receiver/exporter combo for logs pipelines?

Current config setup

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: '0.0.0.0'
                port: 9991
  pipelines:
    logs:
      receivers:
      - netflow
      processors:
      - custom_netflow_processor
      exporters:
      - debug 

[CaptainPickles89]

Hi Shahparthiv,
If you are looking to get the internal collector metrics and export them as metrics, why don't you just use a metrics pipeline to do that? You could do something like:

receivers:
  prometheus/internal:
    config:
      scrape_configs:
        - job_name: 'internal-collector'
          metrics_path: '/metrics'
          scrape_interval: 15s
          static_configs:
            - targets: ['127.0.0.1:8888']

exporters:
  prometheus:
    endpoint: '0.0.0.0:9991'

service:
  telemetry:
    metrics:
      level: basic
      address: 127.0.0.1:8888
  pipelines:
    metrics/internal:
      receivers: [prometheus/internal]
      processors: []
      exporters: [prometheus]

This will then expose the internal metrics you're after on 8888 then scrape them, you can add whatever processing you need, then expose them on your port 9991.

Does this work for your use case?