Getting Data In

Need environment variable for index ?

Silmarillion197
Explorer

Hi,

we have 180+ machines with different services, which send their data using a splunk forwarder to different indexes. To keep this scenario manageable we use a splunk management instance to rollout inputs.conf and outputs.conf on each of these splunk forwarders. This scenario woks fine, as long as the same index is used for all the services and data for all services arrive in that "superindex".

But it is obligatory to seperate the indexes, as data comes from different services (requirement). So I thought we could establish an environment variable on each linux system, which keeps the servicename and then refer to  that servicename as an index in the inputs.conf

For example like so:
[monitor:///var/log/service/*/service.log]
sourcetype = sc:$SERVICENAME:service:log
disabled = 0
index = $SERVICENAME

That way, we could still rollout the inputs.conf using the splunk manager and only have to set up this environment variable $SERVICENAME once on each machine. But it seems that the environment variable isn`t recognized in the inputs.conf, as on Splunk Indexer there is the message

"Search peer splunk-indexer has the following message: Received event for unconfigured/disabled/deleted index=$servicename with source=[...]"
So it seems that the environment variable $servicename was not resolved to the value which was set on the machine.

Is there a different way to make index in inputs.conf flexible for each machine and nevertheless keep the rollout system, or can it be done using environment variables and we did it wrong somehow? 

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Splunk's support of environment variables is extremely limited so stick to $SPLUNK_HOME.  Also, since this sounds like a multi-tenant operation, you should know Splunk doesn't support multi-tenancy.

Can you get your management instance to substitute $SERVICENAME when it writes the config files?

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Splunk's support of environment variables is extremely limited so stick to $SPLUNK_HOME.  Also, since this sounds like a multi-tenant operation, you should know Splunk doesn't support multi-tenancy.

Can you get your management instance to substitute $SERVICENAME when it writes the config files?

---
If this reply helps you, Karma would be appreciated.

Silmarillion197
Explorer

It is not my management. We are using standard splunk forwarder management for rollouts:
https://docs.splunk.com/Documentation/Splunk/8.0.6/Updating/Forwardermanagementoverview

And I`m not aware of a possibility to get those servicenames right from the splunk forwarder management, because this would require a mapping on the splunk forwarder manager that is aware which service belongs to which host. That is far more complicated to manage, than just storing the servicename once as an environment variable on the corresponding server.

Ok, as you are saying that we can`t use an environment variable in the inputs.conf, I think we have to create a scripted solution of our own for rollout, such as ansible script. That rollout script solution would make the splunk forwarder management useless for us.

0 Karma
Get Updates on the Splunk Community!

New Cloud Intrusion Detection System Add-on for Splunk

In July 2022 Splunk released the Cloud IDS add-on which expanded Splunk capabilities in security and data ...

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...