Splunk Forwarders Input.Conf Setting Environment Variable

[ezajac]

I am looking for a way to deploy Splunk Forwarders to a Server Farm. I would like to package the installation and then copy an inputs.conf file to the server after the deployment.

Is there a way to configure an Environment Variable for the Host Name so the computer name is used like %COMPUTERNAME%? I don't want to have to make a manual change for every server in the server farm.

[default]
host = %COMPUTERNAME%

I tried this earlier and the Indexer used %COMPUTERNAME% as the Host Name.

[andrewcg]

Just use $env_var instead of %env_var%. Even though you are on a Windows server, the format of environment variables accepted in Spunk conf file will follow the Linux format of $envvar. So %COMPUTERNAME% becomes $COMPUTERNAME

The better solution is to not set the host in the inputs.conf file. If you do this it will default to the value in the server.conf file. Then you set it there:

[general]
serverName=$COMPUTERNAME

[triest]

I know this is an old post, but I want to disagree about "the better solution is to not set the host in inputs.conf".

There are situations were processes are known and hosts will get renamed. The default way requires that Splunk be handled as a special case on the rename either running a command to tell Splunk to cleanup (designed for gold images), a command to remove $SPLUNK_HOME/etc/system/local/inputs.conf, or a command to update $SPLUNK_HOME/etc/system/local/inputs.conf If you can set it to the environmental variable at install time, then when the rename occurs, you just need a service restart which in the worst case would be the next reboot. Eventually it will fix itself without having to have people go out of their way to handle Splunk as a special case AND if they remember they should go out of their way, its as simple as rebooting. This makes it much easier to scale Splunk.

I would argue that instead of writing out a $SPLUNK_HOME/etc/system/local/inputs.conf and making it impossible for Splunk administrators to set a system default, it would have been much better if Splunk had just set the hostname to COMPUTERNAME (or similar variable) in $SPLUNK_HOME/etc/system/default/inputs.conf so Splunk admins could push overrides. Yes, I have real world examples of where I wish I had this option today; I can't share specifics, but multiple teams manage different types of systems and some times they won't adjust their processes and at some level it makes sense, they can't change their processes around every tool.

That's the difference of a theoretical knowledge of Splunk administration and having years of actually doing it day in and day out.

[andrewcg]

Not sure what your deal is, insults have no place in support forums.

I am sorry I was not more clear in my answer. You said:

I want to disagree about "the better solution is to not set the host in inputs.conf".

Then you explain how setting the host in input.conf is a bad idea. I am not sure your point or if you made a mistake when reading the question or answer. I agree my sentence is confusingly written. This is the short version of my answer again:

Do not set the host name on inputs.conf unless you are overriding a system default.

• Set a default across the server in local/server.conf (only is you want to override the Splunk default of $HOSTNAME)
• Only set per input overrides in the local/inputs.conf file if you need them to be different then the default, for example on Heavy Forwarders

If you examine the default/server.conf file on Linux (Splunk 8.0.3) you see:

[general]
serverName=$HOSTNAME

This way out of the box Splunk will set host to the name of the server, a host name change will be captured by Splunk.

[triest]

@andrewcg,

This way out of the box Splunk will set host to the name of the server, a host name change will be captured by Splunk.

The setting you quoted is in server.conf We're talking about logs coming into Splunk which is controlled by inputs.conf When Splunk first starts, it writes $SPLUNK/etc/system/local/inputs.conf with the following:

[default]
host = WHATEVER_THE_CURRENT_HOSTNAME_IS

When you then rename the box to something else, it will still send logs in as the original host name because once written that file will not be updated unless you manually update it or clear that setting.

I can't recall having done a rename with a version 8 UF, but the Make a universal forwarder part of a host image documentation still states that ./splunk clone-prep-clear-config "clears instance-specific information, such as the server name and GUID, from the forwarder" so I have no reason to believe it has changed.

[andrewcg]

As you said, this is a really old issue. inputs.conf is created if it is missing, and it is configured to the actual local host value on first start after an install or upgrade. If you set your local/inputs.conf file to $decideOnStartup, it should not be overwritten by installs or upgrades. A simple restart does not create the inputs.conf file.

The original question was specifically in regards to packaging the install and deploying to servers, not about imaging a server with the Splunk install on it. As you point out, you will need a different method of configuring a server that was imaged with Splunk on it. We use that method or VDI deployments. Most of our deployments of Splunk are done using Ansible to make sure the proper config is pushed to whatever server needs it. Splunk is installed and configured on the server after first boot, not during the image creation process. Renaming servers is not allowed in our environment, so we do not have problems with renames. Your environment seems more challenging, so I expert you would have to come up with different solutions. Try using $decideOnStartup and see if that reduces some of your pain points.

[sowings]

I don't think it's possible to utilize %COMPUTERNAME% style variables.

However, it's not impossible to get what you want. Typically, when Splunk starts up for the first time, it writes a new inputs.conf in the etc/system/local subdirectory. This inputs.conf contains just a [default] section like you've described above, with the host set to the "discovered" name of the system (so, basically, the value of %COMPUTERNAME%). If you then ship an inputs.conf in a separate app (e.g. "myco_web_inputs") with the input definition you want minus any host value, the host value from the system/local/inputs.conf will shine through, giving you per-host identified log data.