[cmd/opampsupervisor] Add support for "base layer" local config for the Collector

[douglascamata]

Component(s)

cmd/opampsupervisor

Is your feature request related to a problem? Please describe.

I want to provide a base configuration, that can be overwritten by any other configuration source (the current local configuration implement is final and not overwrite-able), that the Supervisor can use to start the Collector without having to interact with any OpAMP backend to get remote configuration.

Describe the solution you'd like

We already got local configuration (proposed in #37886, implemented in #38671) support in the Supervisor's agent.config_files. These files are merged last into all other configuration, making it impossible to overwrite any configuration they do.

While enforcement of certain configuration above everything else is a very valid use case, I would like to propose that we also another use case: local configuration that can be used as a "base layer" on which to merge everything else.

It could be configured like this:

agent:
  base_config_files:
    - starting_config.yaml
  config_files: 
    - enforced_config.yaml

And I believe this "base config" should have the lowest priority in merging. In other words, it should be the very first to be merged into the config.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• cmd/opampsupervisor: @evan-bradley @atoulme @tigrannajaryan

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[tigrannajaryan]

I understand the use case, but I wonder if there is a way to design this differently so that it is easier to understand by the end users and harder to misuse. The proposed approach requires the user to read the docs and understand the merging order, which already is not easy and becomes even more complicated.

[douglascamata]

@tigrannajaryan that's already a problem we have today (see #39931, for instance) and one that I don't think will ever be avoided (but we can handle it better). We need very good documentation and logs regarding configuration merging. If possible we could also add this to the specification.

Part of the improvement I believe is better naming. That's why I propose the name base_config_files for this new configuration. I think it communicates well what it does and it's low priority. It could also be named low_priority_config_files.

The name config_files is bad in comparison, because it doesn't give you any clue about the priority of these configuration files. I would rather rename it to enforced_config_files, critical_config_files, top_level_config_files, high_priority_config_files, or something else on these lines that clearly communicates it overrides everything else.