Need environment variable for index ?

[Silmarillion197]

Hi,

we have 180+ machines with different services, which send their data using a splunk forwarder to different indexes. To keep this scenario manageable we use a splunk management instance to rollout inputs.conf and outputs.conf on each of these splunk forwarders. This scenario woks fine, as long as the same index is used for all the services and data for all services arrive in that "superindex".

But it is obligatory to seperate the indexes, as data comes from different services (requirement). So I thought we could establish an environment variable on each linux system, which keeps the servicename and then refer to  that servicename as an index in the inputs.conf

For example like so:
[monitor:///var/log/service/*/service.log]
sourcetype = sc:$SERVICENAME:service:log
disabled = 0
index = $SERVICENAME

That way, we could still rollout the inputs.conf using the splunk manager and only have to set up this environment variable $SERVICENAME once on each machine. But it seems that the environment variable isn`t recognized in the inputs.conf, as on Splunk Indexer there is the message

"Search peer splunk-indexer has the following message: Received event for unconfigured/disabled/deleted index=$servicename with source=[...]"
So it seems that the environment variable $servicename was not resolved to the value which was set on the machine.

Is there a different way to make index in inputs.conf flexible for each machine and nevertheless keep the rollout system, or can it be done using environment variables and we did it wrong somehow? 

[richgalloway · Accepted solution]

Splunk's support of environment variables is extremely limited so stick to $SPLUNK_HOME.  Also, since this sounds like a multi-tenant operation, you should know Splunk doesn't support multi-tenancy.

Can you get your management instance to substitute $SERVICENAME when it writes the config files?

[Silmarillion197]

It is not my management. We are using standard splunk forwarder management for rollouts:
https://docs.splunk.com/Documentation/Splunk/8.0.6/Updating/Forwardermanagementoverview

And I`m not aware of a possibility to get those servicenames right from the splunk forwarder management, because this would require a mapping on the splunk forwarder manager that is aware which service belongs to which host. That is far more complicated to manage, than just storing the servicename once as an environment variable on the corresponding server.

Ok, as you are saying that we can`t use an environment variable in the inputs.conf, I think we have to create a scripted solution of our own for rollout, such as ansible script. That rollout script solution would make the splunk forwarder management useless for us.