Adding multiple _meta fields from different configuration apps

[cameronjust]

Hi All,

So I'm trying to come up with a solution where all UFs and HFs add new fields to all indexed data for

env_class = the type of server the logs are from ie mailserver, app_server, webserver

env_type = dev, test or prod.

I can do this with an inputs.conf on the forwarder that looks like this

# Ref: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf

# Add default indexed time field for this type of host
[default]

# These fields will be added to all events coming from this host. See README.TXT or fields.conf for how to make these searchable from search head. All varaibles must be on the same line seperated by space
# Note if this config is on an intermediate Heavy Forwarder they will also be applied to all events passing through that forwarder (even cooked data)
_meta=env_class::workstation env_type::prod

[WinEventLog]
_meta=env_class::workstation env_type::prod

[perfmon]
_meta=env_class::workstation env_type::prod

[WinHostMon]
_meta=env_class::workstation env_type::prod

#  Untested but might be required.
# [WinRegMon]
# _meta=env_class::workstation env_type::prod

All the hosts are dynamically created and destroyed with random hostnames hence the need for these additional fields to be added all events coming from each host. So for dashboards monitoring say perfmon the end user can quickly drill down to all the prod webservers.

Now all these additional indexed fields must be all contained in the one _meta line in a config. Which brings me to my dilemma.

I'd like some control over this from the deployment server with say the following server classes with associated apps.

1. Dev Environment App - Sets the env_type=dev for all hosts with *-dev-* in the hostname
2. Prod Environment App - Sets the env_type=prod for all hosts with *-prod-* in the hostname
3. Webserver Class App - Sets the env_class=webserver 
4. Mailserver Class App - Sets the env_class=mailserver
5. etc....

The problem is that the Environment and Class app will override the setting of _meta and only one will get used in the final setup.

ie

cfg_set_env_type_prod/local/inputs.conf

[default]
_meta=env_type::prod

cfg_set_env_class_webserver/local/inputs.conf

[default]
_meta=env_class::webserver

Because each app uses _meta only the cfg_set_env_class_webserver will apply the _meta since it's wins the precedence war with it's app name. So only env_class will be set and env_type will be empty. 

Are there any solutions that anyone can think of?

Since these are UFs we can't use transforms.conf.